Using MSI-X in KMDF driver question

Hi,
I have a PCIE 3.0 device which supported MSI-X interrupt. It has 3 BAR spaces, BAR0 is for DMA descriptors, BAR1 is for MSI-X entry tables and BAR2 is for hardware operation. It supported 8 MSI interrupts. The device supplied a test MSI interrupt register can fire a test interrupt to system.
I need to develop a new driver for it. My driver can be loaded and initialized device now. I have dumped the interrupt type reasource descriptors and can see OS allocated 8 interrupt vector for my driver.
My question is my driver can’t receive any interrupt after I used WdfInterruptCreate function to register ISR for my driver by setting the interrupt test register(this test interrupt is #4 and #5 MSI interrupt). I don’t know if I need to configure more other features for MSI-X interrupt. Do I need to create interrupt object for every MSI-X entry table? If yes, how can I do this?
Thank you very much!

PS. I have confirmed my hardware is OK in function. Because I installed Windriver general PCI driver for it and windriver driver can show 8 interrupt resources. And it can receive interrupt signal when I fired the test registers.

Could anybody please help me?
Thanks

Try to filter interrupt resources so that you have only one test interrupt when your EvtDevicePrepareHardware is called:

https://msdn.microsoft.com/windows/hardware/drivers/wdf/modifying-a-resource-requirements-list

Or, you can also create an interrupt object for each assigned message ID but with the same InterruptIsr. When the InterruptIsr is called, the message ID is passed as a parameter.

Look at these sample (device.c and interrupt.c):

https://github.com/Microsoft/Windows-driver-samples/tree/master/usb/ufxclientsample

Hi, D.T.
Thank you very much for help me again.

But I got a question again. When I call WdfInterruptCreate to create interrupt object I need to to initialize a WDF_INTERRUPT_CONFIG structure. This structure defined in MSDN document is following:
typedef struct _WDF_INTERRUPT_CONFIG {
ULONG Size;
WDFSPINLOCK SpinLock;
WDF_TRI_STATE ShareVector;
BOOLEAN FloatingSave;
BOOLEAN AutomaticSerialization;
PFN_WDF_INTERRUPT_ISR EvtInterruptIsr;
PFN_WDF_INTERRUPT_DPC EvtInterruptDpc;
PFN_WDF_INTERRUPT_ENABLE EvtInterruptEnable;
PFN_WDF_INTERRUPT_DISABLE EvtInterruptDisable;
} WDF_INTERRUPT_CONFIG, *PWDF_INTERRUPT_CONFIG;
But I can see 2 not exist members in this structure was referenced in the sample code as following:
InterruptConfig.InterruptRaw = InterruptResourceRaw;
InterruptConfig.InterruptTranslated = InterruptResourceTranslated;
I don’t know where those 2 parameters found, that is why I don’t know how to register my MSIX interrupt by interrupt resources because there was no place for me to pass MSIX entry table or entry number when I created interrup object.
Thanks

You get the raw and translated values in EvtDevicePreapareHardware

Peter
OSR
@OSRDrivers

On Dec 11, 2016, at 7:36 AM, xxxxx@hotmail.com wrote:

But I got a question again. When I call WdfInterruptCreate to create interrupt object I need to to initialize a WDF_INTERRUPT_CONFIG structure. This structure defined in MSDN document is following:
…typedef struct _WDF_INTERRUPT_CONFIG {
ULONG Size;
WDFSPINLOCK SpinLock;
WDF_TRI_STATE ShareVector;
BOOLEAN FloatingSave;
BOOLEAN AutomaticSerialization;
PFN_WDF_INTERRUPT_ISR EvtInterruptIsr;
PFN_WDF_INTERRUPT_DPC EvtInterruptDpc;
PFN_WDF_INTERRUPT_ENABLE EvtInterruptEnable;
PFN_WDF_INTERRUPT_DISABLE EvtInterruptDisable;
} WDF_INTERRUPT_CONFIG, *PWDF_INTERRUPT_CONFIG;

Where did you see that description, exactly? There are 7 additional members in this structure.

But I can see 2 not exist members in this structure was referenced in the sample code as following:
InterruptConfig.InterruptRaw = InterruptResourceRaw;
InterruptConfig.InterruptTranslated = InterruptResourceTranslated;
I don’t know where those 2 parameters found, that is why I don’t know how to register my MSIX interrupt by interrupt resources because there was no place for me to pass MSIX entry table or entry number when I created interrup object.

Does your editor not have a way for you to search for those symbols? You’ll see they are parameters handed to the EvtDevicePrepareHardware. You pass them straight through without interpreting the contents.

Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Hi? Tim:

Where did you see that description, exactly? There are 7 additional members in this structure.

I see this structure in microsoft MSDN website. And my WDK document also has same structure. All of them have not these 2 member parameters. My WDK version is 7601. Does this version too old that it doesn’t include these parameters?
Where are you see 7 additional members in this structure? Could you please indicate it to me?
Thank you!

This is the microsoft MSDN website contents:
https://msdn.microsoft.com/zh-cn/library/ff552347(v=vs.85).aspx

xxxxx@hotmail.com wrote:

This is the microsoft MSDN website contents:
https://msdn.microsoft.com/zh-cn/library/ff552347(v=vs.85).aspx

The additional members were added in KMDF 1.11. Check the
English-language version of that exact same page:
https://msdn.microsoft.com/en-us/library/windows/hardware/ff552347.aspx


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Hi, Tim:
Yes, my WDK version is too old, it’s KMDF version is 1.09.
So I need to update WDK to 8.0 or 10 to update KMDF 1.11.
But I was confused by the WDK document which stated WDK 7600 supported MSI interrupt totally. And I suspected if WDK 10 can solute my problem. That is registered IMSR for my device to support more interrupt signals.
Thanks

Hello,

If you’re going to use only MSI interrupt, WDK 7660 (KMDF v1.09) is totally
enough.

But taking into account your question about MSI-X interrupt, I can say that
basic support of this kind of interrupt on the framework level implemented
only for NDIS driver.

Correct me if I’m wrong but I’ve checked this several times and I’ve
examined corresponding sources from the WDK collection.

With respect to MSI interrupt, I’ve sorted out with them.

Basically, all that necessary to do, you have correctly prepare the driver
inf-file and in the right way make initialization of interrupt object.
Number of interrupt objects in given case should be equal to the number of
supported MSI vectors.
In other words, you have to allocate multiple instances of interrupt
object, one instance per MSI vector.
After this important step you can see the reaction of the Windows kernel on
you setup in the WinDbg with aid of the following commands:

!wdfkd.wdftmffile c:\WinDDK\7600.16385.1\tools\tracing\amd64\Wdf01009.tmf

!wdflogdump

If everything has been done correctly, you’ll see how system allocate the
number of MSI vectors that you previously pointed in the inf-file.

Within the configuration space of the your device there is a one dedicated
location called MSI Capability Structure.

In this structure there is a 16-bit register MSI Control. Exactly this
register serves for configuration of interrupt system used by the Host PC.
By default, only legacy interrupt (INTx) support is configured. If you
would like to use MSI or MSI-X your have to point this in the inf-file.
By the way, configuration for MSI/MSI-X interrupt in inf-file has not some
differences, it’s almost the same.

In other words, it’s not a matter of OS support, it’s mainly a matter of
capability of your HW.
OS system just provides a mechanism of configuration configuration space of
particular device.
And handling interrupt from this device via PCIe interface.

Necessary just properly configure MSI Control register depends on the
capability of your device.
After the loading of your driver, system will reconfigure the interrupt
system based on the settings in your inf-file.

For MSI-X interrupt system within the configuration space there is another
location called MSI-X Capability Structure.
Here you have also quite the same registers set, namely: Message Control,
Table Offset and PBA(Pending Bit Array) Offset.
Unlike the MSI interrupt, for MSI-X natively (by the design) can be assign
corresponding priority and in which connection, this priority will be
handle by the Root Complex.
In the case of MSI, you also has such functionality but it’s implemented on
the Windows kernel level (allowed for you specify the logical CPU core,
priority and affinty - WdfInterruptSetPolicy())

MSI interrupt support I’ve implemented and tested (8 MSI vectors,
Motherboard based on the Q87 chipset and Haswell CPU with 4 logical cores
without Hyper-Threading feature support, PCIe gen2 x8).

But MSI-X support I guess required corresponding support from the OS
because you have to manage interrupt table register and PBA register. At
least during the configuration stage.
NDIS network driver library provides such APIs. But if you’re develops
ordinary bus driver such APIs are absent.
That’s why I decided to proceed use MSI instead of MSI-X.

Except of prioritizing MSI-X has another important feature - it’s can
support much more (for modern Windows kernel this limit a bit less that
2048) than 32 interrupt sources (it’s limit for MSI by the way).

Certainly most of this information you can take from the PCIe
specification. But still important is support from the OS side.
And with the support of MSI-X it’s still remaining not clear how this is
implemented by KMDF for PCIe bus driver not for NDIS.

Best Regards,
Dmitry

On Tue, Dec 13, 2016 at 3:38 AM, wrote:

> Hi, Tim:
> Yes, my WDK version is too old, it’s KMDF version is 1.09.
> So I need to update WDK to 8.0 or 10 to update KMDF 1.11.
> But I was confused by the WDK document which stated WDK 7600 supported MSI
> interrupt totally. And I suspected if WDK 10 can solute my problem. That is
> registered IMSR for my device to support more interrupt signals.
> Thanks
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: http:> showlists.cfm?list=ntdev>
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:></http:>

Hello,
Thank you very much!
Could you please share some sample code or example for this?

Hi,
One more question: about the MSI-X entry table, it is OS system responsibility or device driver’s responsibility to initialize them?
Thanks

Hi,
Thank you all for the great help for me!!!
After using new wdf 1.11 framework, my device’s MSI-X interrupts can be registered normally. My IMSRs can receive device hardware INT signal now.


PCI 3.0 defines an extended form of MSI, called?MSI-X, that enables greater programmability. Windows?Vista and later versions of Windows support MSI and MSI-X.?

https://msdn.microsoft.com/en-us/library/windows/hardware/ff548079(v=vs.85).aspx

So MSI-X should be available with WDK7.1. Why can’t you use MSI-X with WDK7.1 ?

xxxxx@gmail.com wrote:


> PCI 3.0 defines an extended form of MSI, called?MSI-X, that enables greater programmability. Windows?Vista and later versions of Windows support MSI and MSI-X.?
>

https://msdn.microsoft.com/en-us/library/windows/hardware/ff548079(v=vs.85).aspx

So MSI-X should be available with WDK7.1. Why can’t you use MSI-X with WDK7.1 ?

You can, with WDM. It just wasn’t fully baked into KMDF yet.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.

Hello,

Answering on your question regarding of applying any changes in MSI-X entry
table, I can say following - it should be supports from the both side (OS +
PCIE device). For better understanding of MSI-X interrupt work, I’m
recommending to familiar with following article

http://www.alterawiki.com/wiki/Implementing_MSI-X_for_PCI_Express_in_Altera_FPGA_Devices

This article explain implementation details of MSI-X interrupt controller
using as an example Altera FPGA. But it’s only about the implementation
inside FPGA .

From the OS side, corresponding support should be implemented as well. As
was very good mentioned by xxxxx@gmail.com (unfortunately I
don’t know his name) and Mr. Roberts, now it is only made in the WDM but
absent in KMDF.

I’m thinking exactly about this was your original question.

Therefore possible to provide full support of MSI-X interrupt using WDM
APIs, you can use WDM invocations in your KMDF driver code. I think you’re
know about this.

Please carefully exemine the web-link that was kindly provided by
xxxxx@gmail.com.

Best Regards,
Dmitry

On Dec 27, 2016 20:46, “Tim Roberts” wrote:

xxxxx@gmail.com wrote:
>
> PCI 3.0 defines an extended form of MSI, called?MSI-X, that enables
greater programmability. Windows?Vista and later versions of Windows
support MSI and MSI-X.?
>
>
> https://msdn.microsoft.com/en-us/library/windows/hardware/
ff548079(v=vs.85).aspx
>
> So MSI-X should be available with WDK7.1. Why can’t you use MSI-X with
WDK7.1 ?

You can, with WDM. It just wasn’t fully baked into KMDF yet.


Tim Roberts, xxxxx@probo.com
Providenza & Boekelheide, Inc.


NTDEV is sponsored by OSR

Visit the list online at: http:>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and
software drivers!
Details at http:

To unsubscribe, visit the List Server section of OSR Online at <
http://www.osronline.com/page.cfm?name=ListServer&gt;</http:></http:>

>You can, with WDM. It just wasn’t fully baked into KMDF yet.

Ok, thank you.