MSI-X Vectors

Alex_Chang · February 24, 2010, 12:33pm

Does anyone have experience that the number of MSI-X vectors Windows would enable for a individual device is less than the device supports ?

Thanks a lot,
Alex

Jake_Oshins · February 24, 2010, 1:40pm

Yes. Maybe you could ask a more specific question? Or elaborate on your
general issue?

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Does anyone have experience that the number of MSI-X vectors Windows would
> enable for a individual device is less than the device supports ?
>
> Thanks a lot,
> Alex
>

Alex_Chang · February 24, 2010, 2:37pm

Hi Jake,

Thanks for reply.
I am wondering how Windows would grant the MSI-X vectors for an I/O Adapter device? The device sets up in its MSI-X capability stucture, saying it supports 32 vectors (Table Size) and its miniport driver has the registry entries set up accordingly (MessageNumberLimit = 32, MSISupported = 1). Is it asking too much? If OS can only enable part of them, will the MessageID be contiguous ? For example start from 0 to 15?

I have another question regading NUMA support. In current WDK, most the NUMA node related APIs can only support Windows 7. Let’s say, if I need to allocate contiguous system memory associated with certain nodes via MmAllocateContiguousMemorySpecifyCacheNode on Windows Server 8, do I have any alternatives to do that in my miniport driver ?

Thanks a lot in advance,
Alex

Jake_Oshins · February 24, 2010, 8:15pm

This is all covered in the WDK documentation. Windows will either give you
your entire request or a single message, or a line based interrupt, if no
MSI option exists. You can see this in the debugger if you type “!devnode
pointer-for-your-devnode 6”. You’ll see the resource requirements lists
there that might be assigned to your device. There will be your 32-message
claim, and then a claim involving a single interrupt. Your 32-message claim
will be satisfied if 32 free IDT entries exist and the system itself
supports MSI.

The rest of your question depends on your definition of “miniport.”
Unfortunately, we have storport miniports, NDIS miniports, video miniports,
etc.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> Thanks for reply.
> I am wondering how Windows would grant the MSI-X vectors for an I/O
> Adapter device? The device sets up in its MSI-X capability stucture,
> saying it supports 32 vectors (Table Size) and its miniport driver has the
> registry entries set up accordingly (MessageNumberLimit = 32, MSISupported
> = 1). Is it asking too much? If OS can only enable part of them, will the
> MessageID be contiguous ? For example start from 0 to 15?
>
> I have another question regading NUMA support. In current WDK, most the
> NUMA node related APIs can only support Windows 7. Let’s say, if I need to
> allocate contiguous system memory associated with certain nodes via
> MmAllocateContiguousMemorySpecifyCacheNode on Windows Server 8, do I have
> any alternatives to do that in my miniport driver ?
>
> Thanks a lot in advance,
> Alex
>
>

Alex_Chang · February 25, 2010, 11:26am

Hi Jake,

I am implementing Storport miniport driver and would like to take advantage of NUMA features.

Thanks a lot,
Alex

Jake_Oshins · February 25, 2010, 11:44am

The absolute most important thing you can do is to make sure that you
support MSI-X, with at least one vector per core, which will allow storport
to tell you, using STARTIO_PERFORMANCE_PARAMETERS, which table entry to use.
My guess is that most of the rest of the things that you might do are in the
noise.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> I am implementing Storport miniport driver and would like to take
> advantage of NUMA features.
>
> Thanks a lot,
> Alex
>

Alex_Chang · February 25, 2010, 12:29pm

Hi Jake,

You’re right that it would be great to assoicate one vector with a specific core. Furthermore, if I have the complete topology of NUMA nodes/cores and allocate the separate memory resources associated with different cores, most of the lockings will not be necessary on the fly. Do you think I am trying to do too much that Windows is not yet supported in Storport on Server 2008?

Thanks,
Alex

Jake_Oshins · February 25, 2010, 12:47pm

No, I think that you’re overthinking the problem. What I described is
exactly what storport does for you, even in Windows Server 2008. All you
have to do is support MSI-X with a vector count large enough to map onto all
the cores.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> You’re right that it would be great to assoicate one vector with a
> specific core. Furthermore, if I have the complete topology of NUMA
> nodes/cores and allocate the separate memory resources associated with
> different cores, most of the lockings will not be necessary on the fly. Do
> you think I am trying to do too much that Windows is not yet supported in
> Storport on Server 2008?
>
> Thanks,
> Alex
>

Alex_Chang · February 25, 2010, 1:43pm

I understand that PERF_CONFIGURATION_DATA and STARTIO_PERFORMANCE_PARAMETERS help the mapping between processor cores and MSI-X vectors (if large enough). The concern I have here is the memory locality of NUMA nodes. The performance is much better if I can allocate the memory buffers associated with certain nodes with StorPortAllocateContiguousMemorySpecifyCacheNode or MmAllocateContiguousMemorySpecifyCacheNode. However, they seems only available in Windows 7.

Thanks,
Alex

Jake_Oshins · February 25, 2010, 2:05pm

Yes, that’s true. And you should do that on systems that allow it. But the
locality of the data is much, much more important, as it’s less likely to be
in cache.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> I understand that PERF_CONFIGURATION_DATA and
> STARTIO_PERFORMANCE_PARAMETERS help the mapping between processor cores
> and MSI-X vectors (if large enough). The concern I have here is the memory
> locality of NUMA nodes. The performance is much better if I can allocate
> the memory buffers associated with certain nodes with
> StorPortAllocateContiguousMemorySpecifyCacheNode or
> MmAllocateContiguousMemorySpecifyCacheNode. However, they seems only
> available in Windows 7.
>
> Thanks,
> Alex
>

Alex_Chang · February 25, 2010, 7:59pm

Hi Jake,

Sorry to bother you again. I thought you would know the answer to this:
Is Storport miniport driver responible for reporting the disk size of devices under the adapter controller? If yes, does it get called to report via IOCTL_STORAGE_READ_CAPACITY with STORAGE_READ_CAPACITY? And when ?

Many thanks,
Alex

Jake_Oshins · February 25, 2010, 9:18pm

That’s not my area. I’m sure it’s covered in the WDK.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> Sorry to bother you again. I thought you would know the answer to this:
> Is Storport miniport driver responible for reporting the disk size of
> devices under the adapter controller? If yes, does it get called to report
> via IOCTL_STORAGE_READ_CAPACITY with STORAGE_READ_CAPACITY? And when ?
>
> Many thanks,
> Alex
>

Alex_Grig · February 25, 2010, 10:19pm

The miniport doesn’t handle IOCTLs, other than custom (private).

READ_CAPACITY is handled by disk.sys.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> Sorry to bother you again. I thought you would know the answer to this:
> Is Storport miniport driver responible for reporting the disk size of
> devices under the adapter controller? If yes, does it get called to report
> via IOCTL_STORAGE_READ_CAPACITY with STORAGE_READ_CAPACITY? And when ?
>
> Many thanks,
> Alex
>

Maxim_S_Shatskih · February 25, 2010, 11:43pm

SCSIOP_READ_CAPACITY CDB is the thing.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> Sorry to bother you again. I thought you would know the answer to this:
> Is Storport miniport driver responible for reporting the disk size of devices under the adapter controller? If yes, does it get called to report via IOCTL_STORAGE_READ_CAPACITY with STORAGE_READ_CAPACITY? And when ?
>
> Many thanks,
> Alex
>

Alex_Grig · February 26, 2010, 9:53am

But he would just be passing it down to the target, like any other CDB.

“Maxim S. Shatskih” wrote in message
news:xxxxx@ntdev…
SCSIOP_READ_CAPACITY CDB is the thing.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> Sorry to bother you again. I thought you would know the answer to this:
> Is Storport miniport driver responible for reporting the disk size of
> devices under the adapter controller? If yes, does it get called to report
> via IOCTL_STORAGE_READ_CAPACITY with STORAGE_READ_CAPACITY? And when ?
>
> Many thanks,
> Alex
>

Alex_Chang · February 26, 2010, 11:38am

If SCSIOP_READ_CAPACITY is the one, I need to interpret it and respond it with proper information.
Many thanks to you all.

Alex

Alex_Chang · March 4, 2010, 11:45am

Hi Jake,

I have quick questions regarding MSI-X resource configuration by OS:

When the vectors are granted by the OS, before or after DriverEntry gets called ? I mean the addresses and data are written in vectors.
My hardware has MSI-X Cap Structure implemented and requesting 32 vectors, and doesn’t not support INTx, driver’s INF has all MSI related hardware keys set up, too. However, OS returns STOR_STATUS_INVALID_DEVICE_REQUEST when I call StorPortGetMSIInfo, what are the possible causes to that ? Here are my INF set up:
[idtnvmhci_Inst.HW]
AddReg = idtnvmhci_Inst_MSI_AddReg
[idtnvmhci_Inst_MSI_AddReg]
HKR,“Interrupt Management\MessageSignaledInterruptProperties”, MSISupported, 0x00010001, 1
HKR,“Interrupt Management\MessageSignaledInterruptProperties”, 0x00000010
HKR,“Interrupt Management\MessageSignaledInterruptProperties”, MessageNumberLimit, 0x00010001, 32
HKR,“Interrupt Management\Affinity Policy”, 0x00000010
HKR,“Interrupt Management\Affinity Policy”, DevicePolicy, 0x00010001, 5

Many thanks,
Alex

Jake_Oshins · March 4, 2010, 1:15pm

As I said before, you’d have to ask somebody who knows storport inside and
out. It’s wrapping the OS.

To answer your specific general question, vectors are assigned after
IRP_MN_FILTER_RESOURCE_REQUIREMENTS completes.

–
Jake Oshins
Hyper-V I/O Architect
Windows Kernel Group

This post implies no warranties and confers no rights.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> I have quick questions regarding MSI-X resource configuration by OS:
> 1. When the vectors are granted by the OS, before or after DriverEntry
> gets called ? I mean the addresses and data are written in vectors.
> 2. My hardware has MSI-X Cap Structure implemented and requesting 32
> vectors, and doesn’t not support INTx, driver’s INF has all MSI related
> hardware keys set up, too. However, OS returns
> STOR_STATUS_INVALID_DEVICE_REQUEST when I call StorPortGetMSIInfo, what
> are the possible causes to that ? Here are my INF set up:
> [idtnvmhci_Inst.HW]
> AddReg = idtnvmhci_Inst_MSI_AddReg
> [idtnvmhci_Inst_MSI_AddReg]
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”,
> MSISupported, 0x00010001, 1
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”, 0x00000010
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”,
> MessageNumberLimit, 0x00010001, 32
> HKR,“Interrupt Management\Affinity Policy”, 0x00000010
> HKR,“Interrupt Management\Affinity Policy”, DevicePolicy, 0x00010001, 5
>
> Many thanks,
> Alex
>

Alex_Chang · March 4, 2010, 3:18pm

Thanks a lot, Jake.

Alex

Alex_Grig · March 4, 2010, 11:09pm

Please note, that like many other INF writers, you copy a mistake from the
WDK documentation.

These lines:

HKR,“Interrupt Management\MessageSignaledInterruptProperties”, 0x00000010
HKR,“Interrupt Management\Affinity Policy”, 0x00000010

Were supposed to only create these registry keys. Unfortunately, they miss a
comma, so they just create empty registry values named 0x00000010.

Fortunately, setupapi creates those keys for you implicitly. So these lines
are not necessary at all.

wrote in message news:xxxxx@ntdev…
> Hi Jake,
>
> I have quick questions regarding MSI-X resource configuration by OS:
> 1. When the vectors are granted by the OS, before or after DriverEntry
> gets called ? I mean the addresses and data are written in vectors.
> 2. My hardware has MSI-X Cap Structure implemented and requesting 32
> vectors, and doesn’t not support INTx, driver’s INF has all MSI related
> hardware keys set up, too. However, OS returns
> STOR_STATUS_INVALID_DEVICE_REQUEST when I call StorPortGetMSIInfo, what
> are the possible causes to that ? Here are my INF set up:
> [idtnvmhci_Inst.HW]
> AddReg = idtnvmhci_Inst_MSI_AddReg
> [idtnvmhci_Inst_MSI_AddReg]
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”,
> MSISupported, 0x00010001, 1
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”, 0x00000010
> HKR,“Interrupt Management\MessageSignaledInterruptProperties”,
> MessageNumberLimit, 0x00010001, 32
> HKR,“Interrupt Management\Affinity Policy”, 0x00000010
> HKR,“Interrupt Management\Affinity Policy”, DevicePolicy, 0x00010001, 5
>
> Many thanks,
> Alex
>