MSIs: documentation discrepancies regarding

NTDEV
1

I was reading the WDF book, and on page 545 there is a note:
“Important: Your driver’s interrupt service callback must be written so that it can share interrupt vectors or MSIs with another device…”

That didn’t sound right to me, since I thought I had read that one of the major advantages of MSI was that they did not have to be shared. But, I couldn’t remember where I read that, and after a little searching and found a thread on the NTDEV list (http://www.osronline.com/showThread.cfm?link=103738) in which Jake Oshins explicitly says that MSIs *can* be shared, and you have to code your ISR to handle that fact. Fair enough, that’s not too hard, though not as convenient as I thought MSIs would make it for me. I chalked it up to my imagination/wishful thinking.

Then, earlier today I was looking through the “Interrupt Architecture Enhancements in Microsoft Windows Vista”, and right there under the heading Advantages of Using Message-Signaled Interrupts it says:

"Because the number of potentially available MSIs is limited only to the number of interrupt dispatch table (IDT) entries that are available on a system, devices that implement MSIs do not have to share interrupts with other devices. As a result, interrupt latency may be lower and less subject to extreme variance. Interrupt processing overhead, as measured on a system-wide basis, is also lower. In addition, using MSI or MSI-X avoids the problems that are sometimes seen when a device is forced to share an interrupt with a driver that has design or coding defects in its ISR. Thus, using MSIs contributes to both enhanced system performance and greater overall system reliability.

Some devices use MSIs as an alternative to the use of line-based interrupts. As described previously, this ensures that the device does not have to share an interrupt with another device."

So, I’m *not* crazy for thinking that using MSIs meant that my device would not share interrupts with other devices… it says so right there in the documentation for using MSI with Windows Vista. I realize that this is probably just an example of documentation not evolving with the product, but I thought I’d point it out so that someone can update it so as to prevent the further spread of misinformation.

As an aside, I realize that even IDT entries are relatively scarce, so there is always the possibility that there will be more devices requesting MSIs than are available, but I had assumed that if there aren’t enough MSI resources to go around, then some devices would get (exclusive) MSI resources, and the rest would be stuck with a (shared) INTx resource, especially since they have to handle the INTx case anyway.

Thanks!
-phil

2

The wording in the whitepaper is a bit too aggressive, sharing, while highly unlikely, can still occur if the OS runs out of IDT entries. The whitepaper will be updated to reflect this.

Thanks for pointing this out.
d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of phil.hindman@ni.com
Sent: Wednesday, October 10, 2007 12:02 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] MSIs: documentation discrepancies regarding

I was reading the WDF book, and on page 545 there is a note:
“Important: Your driver’s interrupt service callback must be written so that it can share interrupt vectors or MSIs with another device…”

That didn’t sound right to me, since I thought I had read that one of the major advantages of MSI was that they did not have to be shared. But, I couldn’t remember where I read that, and after a little searching and found a thread on the NTDEV list (http://www.osronline.com/showThread.cfm?link=103738) in which Jake Oshins explicitly says that MSIs *can* be shared, and you have to code your ISR to handle that fact. Fair enough, that’s not too hard, though not as convenient as I thought MSIs would make it for me. I chalked it up to my imagination/wishful thinking.

Then, earlier today I was looking through the “Interrupt Architecture Enhancements in Microsoft Windows Vista”, and right there under the heading Advantages of Using Message-Signaled Interrupts it says:

"Because the number of potentially available MSIs is limited only to the number of interrupt dispatch table (IDT) entries that are available on a system, devices that implement MSIs do not have to share interrupts with other devices. As a result, interrupt latency may be lower and less subject to extreme variance. Interrupt processing overhead, as measured on a system-wide basis, is also lower. In addition, using MSI or MSI-X avoids the problems that are sometimes seen when a device is forced to share an interrupt with a driver that has design or coding defects in its ISR. Thus, using MSIs contributes to both enhanced system performance and greater overall system reliability.

Some devices use MSIs as an alternative to the use of line-based interrupts. As described previously, this ensures that the device does not have to share an interrupt with another device."

So, I’m *not* crazy for thinking that using MSIs meant that my device would not share interrupts with other devices… it says so right there in the documentation for using MSI with Windows Vista. I realize that this is probably just an example of documentation not evolving with the product, but I thought I’d point it out so that someone can update it so as to prevent the further spread of misinformation.

As an aside, I realize that even IDT entries are relatively scarce, so there is always the possibility that there will be more devices requesting MSIs than are available, but I had assumed that if there aren’t enough MSI resources to go around, then some devices would get (exclusive) MSI resources, and the rest would be stuck with a (shared) INTx resource, especially since they have to handle the INTx case anyway.

Thanks!
-phil

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

VERY interesting question, Phil. Thanks for raising it.

If Jake said MSIs can be shared, they can be shared. After all, he’s the one who designed and wrote the Windows MSI code.

Then again, he’s also the one who explicitly reviewed and approved the “Interrupt Architecture Enhancements in Microsoft Windows Vista” whitepaper – the original version of which I wrote for WinHEC back in New Orleans. That paper was indeed written a long time ago, way before any MSI code shipped, so the code could certainly have evolved since that time.

I’m VERY surprised to hear that MSIs can be shared, and it sounds like a distinctly bad idea to me. I certainly know that I can request X MSIs and be granted significantly less than X. Or that I can request MSI and be given an LBI.

But SHARE my MSIs?

I’d like to know (a lot) more about the specific circumstances under which an MSI can be shared, and how I would actually detect this. One of the “big wins” for MSI is that it can offload the first level of decision making from my ISR to my device, thus allowing my driver to take decisions in its ISR (to queue a DPC, for example) based on message number and without even touching my hardware registers.

Does the possibility that MSIs can be shared mean that I need to check my hardware during every interrupt to see if INTREQ is asserted? If that’s the case, and if sharing MSIs is really rare, doesn’t that mean that I pay the penalty for touching my registers during every interrupt to support a rare case?

The way I thought it worked was if you couldn’t get ANY exclusive MSIs then you got an LBI. Wouldn’t that be preferable to having to check your hardware during every MSI interrupt.

Inquring minds, and all that,

Peter
OSR

4

xxxxx@osr.com wrote:

But SHARE my MSIs?

I ended up being bitten by this. I encountered a “funny” on PnP
rebalance testing, which led me to post a query here, as to why my ISR
was being called repeatedly, and Jake replied:

Remember the protocol for shared interrupts. It differs for edge- and
level-triggered interrupts. For level-triggered interrupts, you can
guarantee that you never miss calling the correct ISR if you dismiss the
interrupt at the local APIC after you find one ISR that returns “TRUE” or
when you hit the end of the ISR chain. If the interrupt is still asserted,
then the I/O APIC will pick it up again and send another message to the
processor telling the processor to jump back through the Interrupt
Descriptor Table (IDT) and you’ll run the ISR chain again.

But with edge-triggered interrupts, the protocol is more complicated.
You’ll never get another interrupt if you fail to call the right ISR at the
right time. So the NT kernel calls every ISR in the chain (when an
edge-triggered vector is shared) repeatedly until every single ISR in the
chain returns FALSE. This guarantees that an interrupt that comes in while
the ISR chain is being run won’t be masked by an EOI.

This led me to wondering what situations led to MSI’s being shared:

There are several “ideas” in MSI. MSI means that you don’t have to share
IRQs, as you’re not using them. But IDT entries are still a somewhat scarce
resource, and you may have to share them. This is what the original poster
was testing.

Another important idea with MSI is that you don’t have to touch your
hardware (if the hardware is usefully designed) in your ISR because MSIs
always follow DMA transactions through the chipset so you can guarantee that
your in-memory data is coherent by the time your ISR is invoked.

HTH,

MH.

5

Yes, yes… I understand edge vs level triggered semantics. That has to do with how the ISRs are called by KiInterruptDispatch… All very nice, but sort of beside the point. The issue really is:

THIS relates directly to my question.

I feel like I’m missing something fundamental. I get into my ISR I get message number 256. Okey dokey. Do I now have to determine if it’s that MY message 256?? And I can do this without touching my hardware??

Peter
OSR

6

Peter,

The issue really is:

THIS relates directly to my question.

I feel like I’m missing something fundamental. I get into my ISR I get message
number 256. Okey dokey. Do I now have to determine if it’s that MY message
256?? And I can do this without touching my hardware??

In my opinion, the quote that you have provided gives a very precise answer to your question.In my opinion, the above quote says approx. the following:

You don’t have to access your hardware in ISR in order to check whether MSI is yours -
all that your ISR has to do is check *in-memory* data that had been put into memory by your device’s DMA engine. Once MSI gets raised by writing to Message Data Register, which is done directly by your device as DMA operation, in-memory data is guaranteed to be coherent by the time your ISR is invoked…

Therefore, even if MSI shares interrupt vector, you can always discover whether interrupt is yours without going anywhere close to the hardware - if you had to access the your device in order to check whether it has interrupted, it would just defeat the very purpose of MSI, in the first place…

Anton Bassov

7

Okay, here’s the story. I apologize that the documentation is inconsistent, and that I had Doron reply to this thread earlier with bad information.

MSIs are not shareable. This is definitely true on Vista. Early on, in Jake’s initial code, we allowed sharing. But the problems that have been brought up here, in addition to weird implementation vagaries like that a 2048 message device would end up with 2048 shared MSIs instead of 1 exclusive one, caused us to turn it off.

The documentation was left as it is to provide flexibility for the future, and because as Anton says, it is possible to check software state to determine whether an interrupt has happened. Hardware accesses theoretically shouldn’t be necessary to properly respond TRUE or FALSE to a shared MSI. I remembered the documentation, and mis-remembered the code history, when giving Doron the info he responded with earlier.

While it’s theoretically possible to handle things in the way the WDF book says, it pushes a lot more complexity onto the ISR, and maybe isn’t even possible in some hardware designs. Given that we’ve already shipped an OS that explicitly doesn’t share, I don’t see how we could ever ship one in the future that does. This means that the MSI whitepaper is accurate, and we’ll add this to the WDF book errata list. MSIs can’t be shared.

Martin, I believe your issue was that you had 4 ISRs that all connected to the same vector. It was one MSI allocation, but since your device supported 4 messages you had 4 ISRs, and when the device fell back to 1 message you connected all 4 ISRs to that vector. In this case, sharing can exist, and the advice that Jake gave for avoiding infinitely looping chained interrupts still applies. But that case is “sharing” of an IDT entry by ISRs that are all responding to the same hardware message. Two different devices will never be assigned to the same vector.

Hope that clears things up.

Dave

8

Thanks, Dave, for taking the time to check the code and answer. I think that turning off MSI sharing was a good decision. I know that *I’D* rather fall-back to my LBI than have to think about sharing my MSIs.

Horray! I feel much better now.

Peter
OSR

9

Just one more point:

Your driver will (by default) fall back to one MSI, not the LBI, if there
are not enough IDT entries to share. If you don’t like this behavior and
you want it to fall back directly to the LBI, you have only to filter your
resource requirements list and remove the entry for one MSI.

  • Jake

wrote in message news:xxxxx@ntdev…
> Thanks, Dave, for taking the time to check the code and answer. I think
> that turning off MSI sharing was a good decision. I know that I’D
> rather fall-back to my LBI than have to think about sharing my MSIs.
>
> Horray! I feel much better now.
>
> Peter
> OSR
>

10

You’ll fall back to one EXCLUSIVE MSI, assuming that’s all that’s available, right??

Is it not at least theoretically possible that there are NO EXCLUSIVE interrupt vectors available? In which case, you’d fall back to a (possibly shared) LBI, no??

Perhaps that strains the bounds of what’s likely, but that’s the situation I was thinking about :wink:

Peter
OSR

11

Yes. I was only trying to say that there’s an intermediate position.

  • Jake

wrote in message news:xxxxx@ntdev…
>


>
> You’ll fall back to one EXCLUSIVE MSI, assuming that’s all that’s
> available, right??
>
> Is it not at least theoretically possible that there are NO EXCLUSIVE
> interrupt vectors available? In which case, you’d fall back to a
> (possibly shared) LBI, no??
>
> Perhaps that strains the bounds of what’s likely, but that’s the situation
> I was thinking about :wink:
>
> Peter
> OSR
>
>
>
>