Maximum KMDF Interrupts - Windows 7 x64

NTDEV
1

Hi,

I am working with an FPGA which is connected via PCIe to the computer. The firmware that I have designed for the device relies on MSI (interrupts) to perform tasks. At the moment each card uses 5 MSI vectors which is working fine - and well within the 2048/device limit.

However the FPGA is not intended to be operated alone - in fact in the end there will be up to 16 cards in the system - connected through a pair of PCIe backplanes.

I have got to the stage where I am starting to rack up multiple card and run into a problem. When there are 6 or fewer cards in the system, everything works fine - the KMDF driver correctly loads for each card and calls the WDF interrupt enable function for each of the 5 MSI interrupts. But when I connect the 7th card into the system, only the first 6 have their interrupts enabled - the 7th card only is allocated a single interrupt.

By my calculation, this means that Windows is failing to allocate interrupts if there are more than 31 interrupts in the system. I can’t see anywhere in any documentation why this limit exists!

I may be able to redesign the firmware for the FPGA to reduce the required interrupts down to 2 hopefully without impacting performance too much. But is even if I do that, in a full 16 card system that would require 32 interrupts in total - which the previous test seems to indicate is still too many. Going down to a single vector would be difficult in the system topology, as one is required by an application layer in the FPGA firmware, and at least one is required in the core part of the firmware.

Can anyone shed any light on how to get around this problem?

2

Because MSI vectors are allocated in powers of two, I’d expect a big difference between asking for 5 (which is really asking for 8) and asking for 2 (which asking for, ah, 2).

But it’s hard to know what’s available.

You could also consider using MSI-X as an alternative.

Peter
OSR
@OSRDrivers

3

thomas.carpenter@me.gatech.edu wrote:

I am working with an FPGA which is connected via PCIe to the computer. The firmware that I have designed for the device relies on MSI (interrupts) to perform tasks. At the moment each card uses 5 MSI vectors which is working fine - and well within the 2048/device limit.
…
I may be able to redesign the firmware for the FPGA to reduce the required interrupts down to 2 hopefully without impacting performance too much. But is even if I do that, in a full 16 card system that would require 32 interrupts in total - which the previous test seems to indicate is still too many. Going down to a single vector would be difficult in the system topology, as one is required by an application layer in the FPGA firmware, and at least one is required in the core part of the firmware.

As far as I know, there is no guarantee in PCIExpress that you can get
more than one MSI per endpoint. You can ASK for more, but the system is
not obligated to assign them.

I don’t understand why there should be a painful architectural
difference between having 5 MSIs, and having one MSI with a bit-vector
register indicating which interrupt(s) have fired. The ISR handling is
virtually the same as well.

–
Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.