Problem Setting Interrupt Affinity

Hi

I am writing driver for pcie device for Windows 2008 R2 - SP1. I am using Intel Xeon x64 machine which supports two NUMA nodes with 16 processors each.

I am seeing intermittent performance degradation for my application where my device generates lots of interrupts.
From xperf and kernrate, I can consistently observe that the performance is good when all interrupts are directed to Node 0, whereas when performance is bad all interrupts were directed to Node 1.

From IoGetDeviceNumaNode() I could find that my device is close to node 0. Therefore, I am trying to set affinity for my device interrupts so that all interrupts would be directed to node 0.

I tried using EvtDeviceFilterRemoveResourceRequirements as below -

switch (descriptor->Type) {
case CmResourceTypeInterrupt:
descriptor->u.Interrupt.AffinityPolicy = IrqPolicyAllCloseProcessors;
descriptor->u.Interrupt.PriorityPolicy = IrqPriorityHigh;
descriptor->Flags |= CM_RESOURCE_INTERRUPT_POLICY_INCLUDED;
}

But I am still seeing interrupts intermittently directed to Node 1 processors. I also tried using IrqPolicySpecifiedProcessors and setting TargetedProcessors to affinity mask for Node 0.

Also tried adding following entry in hardware section for my device inf (in addition to MSI entries) -
HKR,Interrupt Management\Affinity Policy,DevicePolicy,0x00010001,1

Another thing I tried was to use Interrupt Affinity Tool to set the same properties for my device.

With all these, intermittently all the interrupts are being directed to processors in Node 1.

I am making some trial mistake here? Please help.
Thanks in advanced.

Thanks & Regards,
Kailas

Did you try *WdfInterruptSetPolicy et al?*

Mark Roddy

On Thu, Jul 24, 2014 at 10:12 AM, wrote:

> Hi
>
> I am writing driver for pcie device for Windows 2008 R2 - SP1. I am using
> Intel Xeon x64 machine which supports two NUMA nodes with 16 processors
> each.
>
> I am seeing intermittent performance degradation for my application where
> my device generates lots of interrupts.
> From xperf and kernrate, I can consistently observe that the performance
> is good when all interrupts are directed to Node 0, whereas when
> performance is bad all interrupts were directed to Node 1.
>
> From IoGetDeviceNumaNode() I could find that my device is close to node 0.
> Therefore, I am trying to set affinity for my device interrupts so that all
> interrupts would be directed to node 0.
>
> I tried using EvtDeviceFilterRemoveResourceRequirements as below -
>
> switch (descriptor->Type) {
> case CmResourceTypeInterrupt:
> descriptor->u.Interrupt.AffinityPolicy =
> IrqPolicyAllCloseProcessors;
> descriptor->u.Interrupt.PriorityPolicy = IrqPriorityHigh;
> descriptor->Flags |= CM_RESOURCE_INTERRUPT_POLICY_INCLUDED;
> }
>
> But I am still seeing interrupts intermittently directed to Node 1
> processors. I also tried using IrqPolicySpecifiedProcessors and setting
> TargetedProcessors to affinity mask for Node 0.
>
> Also tried adding following entry in hardware section for my device inf
> (in addition to MSI entries) -
> HKR,Interrupt Management\Affinity Policy,DevicePolicy,0x00010001,1
>
> Another thing I tried was to use Interrupt Affinity Tool to set the same
> properties for my device.
>
> With all these, intermittently all the interrupts are being directed to
> processors in Node 1.
>
> I am making some trial mistake here? Please help.
> Thanks in advanced.
>
> Thanks & Regards,
> Kailas
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Hi Mark,

My driver is creating Interrupt in PrepareHardware handler. Therefore, I could not try WdfInterruptSetPolicy as specified in it’s doc at http://msdn.microsoft.com/en-us/library/windows/hardware/ff547387(v=vs.85).aspx

Thanks & Regards,
Kailas

Why not create it in EvtDriverDeviceAdd?

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@yahoo.com
Sent: Thursday, July 24, 2014 10:04 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Problem Setting Interrupt Affinity

Hi Mark,

My driver is creating Interrupt in PrepareHardware handler. Therefore, I could not try WdfInterruptSetPolicy as specified in it’s doc at http://msdn.microsoft.com/en-us/library/windows/hardware/ff547387(v=vs.85).aspx

Thanks & Regards,
Kailas


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Okay. I will try if I can create interrupt in EvtDriverDeviceAdd.
I need to allocate IoMemory for bar space. Therefore, I am performing all resource allocations in PrepareHardware. I am new to Windows driver writing. Therefore, I am not sure whether it would be feasible to create interrupt in AddDevice and allocate IoMemory in PrepareHardware.
I will check and get back to you.

However, I am confused on why all other documented ways for interrupt affinity setting are not working.

Thanks & Regards,
Kailas

Hi Mark, Doron,

I did interrupt object creation in AddDevice and used WdfInterruptSetExtendedPolicy() as below -

numa_node = 0; // Got it from IoGetDeviceNumaNode
affinity = 0xF; //
WDF_INTERRUPT_EXTENDED_POLICY_INIT(&interruptPolicy);
interruptPolicy.Priority = WdfIrqPriorityHigh;
interruptPolicy.Policy = WdfIrqPolicySpecifiedProcessors;
interruptPolicy.TargetProcessorSetAndGroup.Group = numa_node;
interruptPolicy.TargetProcessorSetAndGroup.Mask = affinity;
WdfInterruptSetExtendedPolicy(
wam_dev->interrupt,
&interruptPolicy
);

However, I am still seeing large number of interrupts being assigned to other processors.
I tried other options such as WdfIrqPolicyOneCloseProcessor, WdfIrqPolicyAllCloseProcessors etc.
Also tried setting only one processor with WdfIrqPolicySpecifiedProcessors which worked sometime intermittently.

I newly discovered that even though (Task Manager->Process Affinity) section shows two Numa nodes, kernel shows only one processor group. It looks like for processors < 64, only one group would be created. I do not understand how performance tuning is done for such Numa machines. Is it recommended to use ‘groupsize’ setting on these machines?

I also tried creating two groups using ‘groupsize’ and setting affinity to one group. No success.

Thanks & Regards,
Kailas

Groups != numa nodes. Groups is a way to express affinity after you reach the limits of a quad word (64 individual bits). Does !wdfkd.wdflogdump after pnp start say anything useful wrt interrupt assignment?

d

Bent from my phone


From: xxxxx@yahoo.commailto:xxxxx
Sent: ?7/?25/?2014 6:12 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Problem Setting Interrupt Affinity

Hi Mark, Doron,

I did interrupt object creation in AddDevice and used WdfInterruptSetExtendedPolicy() as below -

numa_node = 0; // Got it from IoGetDeviceNumaNode
affinity = 0xF; //
WDF_INTERRUPT_EXTENDED_POLICY_INIT(&interruptPolicy);
interruptPolicy.Priority = WdfIrqPriorityHigh;
interruptPolicy.Policy = WdfIrqPolicySpecifiedProcessors;
interruptPolicy.TargetProcessorSetAndGroup.Group = numa_node;
interruptPolicy.TargetProcessorSetAndGroup.Mask = affinity;
WdfInterruptSetExtendedPolicy(
wam_dev->interrupt,
&interruptPolicy
);

However, I am still seeing large number of interrupts being assigned to other processors.
I tried other options such as WdfIrqPolicyOneCloseProcessor, WdfIrqPolicyAllCloseProcessors etc.
Also tried setting only one processor with WdfIrqPolicySpecifiedProcessors which worked sometime intermittently.

I newly discovered that even though (Task Manager->Process Affinity) section shows two Numa nodes, kernel shows only one processor group. It looks like for processors < 64, only one group would be created. I do not understand how performance tuning is done for such Numa machines. Is it recommended to use ‘groupsize’ setting on these machines?

I also tried creating two groups using ‘groupsize’ and setting affinity to one group. No success.

Thanks & Regards,
Kailas


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

So he needs to call
VOID
KeQueryNodeActiveAffinity(
__in USHORT NodeNumber,
__out_opt PGROUP_AFFINITY Affinity,
__out_opt PUSHORT Count
);

to get the correct affinity mask for the numa node of interest? Would just
using IrqPolicyAllCloseProcessors in the call to
WdfInterruptSetExtendedPolicy be simpler and just as effective?

Mark Roddy

On Fri, Jul 25, 2014 at 9:57 AM, Doron Holan
wrote:

> Groups != numa nodes. Groups is a way to express affinity after you
> reach the limits of a quad word (64 individual bits). Does
> !wdfkd.wdflogdump after pnp start say anything useful wrt interrupt
> assignment?
>
> d
>
> Bent from my phone
> ------------------------------
> From: xxxxx@yahoo.com
> Sent: ‎7/‎25/‎2014 6:12 AM
>
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] Problem Setting Interrupt Affinity
>
> Hi Mark, Doron,
>
> I did interrupt object creation in AddDevice and used
> WdfInterruptSetExtendedPolicy() as below -
>
> numa_node = 0; // Got it from IoGetDeviceNumaNode
> affinity = 0xF; //
> WDF_INTERRUPT_EXTENDED_POLICY_INIT(&interruptPolicy);
> interruptPolicy.Priority = WdfIrqPriorityHigh;
> interruptPolicy.Policy = WdfIrqPolicySpecifiedProcessors;
> interruptPolicy.TargetProcessorSetAndGroup.Group = numa_node;
> interruptPolicy.TargetProcessorSetAndGroup.Mask = affinity;
> WdfInterruptSetExtendedPolicy(
> wam_dev->interrupt,
> &interruptPolicy
> );
>
> However, I am still seeing large number of interrupts being assigned to
> other processors.
> I tried other options such as WdfIrqPolicyOneCloseProcessor,
> WdfIrqPolicyAllCloseProcessors etc.
> Also tried setting only one processor with WdfIrqPolicySpecifiedProcessors
> which worked sometime intermittently.
>
> I newly discovered that even though (Task Manager->Process Affinity)
> section shows two Numa nodes, kernel shows only one processor group. It
> looks like for processors < 64, only one group would be created. I do not
> understand how performance tuning is done for such Numa machines. Is it
> recommended to use ‘groupsize’ setting on these machines?
>
> I also tried creating two groups using ‘groupsize’ and setting affinity to
> one group. No success.
>
> Thanks & Regards,
> Kailas
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev
>
> OSR is HIRING!! See http://www.osr.com/careers
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>

Hi Doron,

Thanks for the clarification on distinction between Nodes and Groups.

I tried !wdfkd.wdflogdump as per your suggestion.
Following is the message wrt interrupts -
FxInterrupt::AssignResources - Is MSI? 1, MSI-ID 0, AffinityPolicy WdfIrqPolicyAllCloseProcessors, Priority WdfIrqPriorityHigh, Group 0, Affinity 0xf, Irql 0x5, Vector 0x51

I am not sure whether this message is for default policy/affinity being assigned when interrupt is created, or for WdfInterruptSetExtendedPolicy() call. Since it’s showing WdfIrqPolicyAllCloseProcessors as policy I suspect that might is for WdfInterruptSetExtendedPolicy().

In order to verify this, I tried reading back Interrupt Info before and after setting policy as below-

RtlZeroMemory(&interruptInfo, sizeof(interruptInfo));
WdfInterruptGetInfo(my_dev->interrupt, &interruptInfo);
KdPrint(“Interrupt Priority:%d Policy:%d Group:%d Affinity:%llx\n”,
interruptInfo.Polarity, interruptInfo.Group, interruptInfo.TargetProcessorSet);

WDF_INTERRUPT_EXTENDED_POLICY_INIT(&interruptPolicy);
interruptPolicy.Priority = WdfIrqPriorityHigh;
interruptPolicy.Policy = WdfIrqPolicyAllCloseProcessors;
WdfInterruptSetExtendedPolicy(
my_dev->interrupt,
&interruptPolicy
);
RtlZeroMemory(&interruptInfo, sizeof(interruptInfo));
WdfInterruptGetInfo(my_dev->interrupt, &interruptInfo);
KdPrint(“New Interrupt Priority:%d Policy:%d Group:%d Affinity:%llx\n”,
interruptInfo.Polarity, interruptInfo.Group, interruptInfo.TargetProcessorSet);

Ouput was as below -

Interrupt Priority:0 Policy:0 Group:0 Affinity:fffffa8000000000
New Interrupt Priority:0 Policy:0 Group:0 Affinity:ffff000200000000

As it can be seen, these values are completely mismatch wrt wdflogdump output.
If these values are correct, then WdfInterruptSetExtendedPolicy() is failing silently.

Can someone please explain what could be the issue?

Thanks & Regards,
Kailas

I think the ‘all close processors’ setting is the default.


Bruce

On Jul 25, 2014, at 10:04 AM, Mark Roddy wrote:
>
> Would just using IrqPolicyAllCloseProcessors in the call to WdfInterruptSetExtendedPolicy be simpler and just as effective?
>