|
RE: ISR Latency question
<![CDATA[I hesitate to respond to this, since I imagine that it will suck me into
a never ending discussion. But if anybody follows your advice, there
will be even more device drivers in the world that have little,
unexplained, odd behaviors. I'll try to break it down point by point.
1. If your device is on either a PCI bus or a bus that mimics the PCI
protocol, then there may be other devices that are hard-wired to share
interrupts with your device. The PCI spec says only that, when your
device wants to generate an interrupt, it should ground one of the INTx#
pins. It's up to the motherboard designer to determine whether the INTx
pins are wired together or whether they each connect to a distinct input
on the interrupt controller. Most low-end motherboards use the 8259 PIC
interrupt controller, which makes them interrupt-constrained. In
practice, this means that your device will share interrupts unless you
choose the motherboard carefully.
Furthermore, if you connect with "exclusive" chosen, one of three things
will happen:
A) The motherboard has already guaranteed you exclusivity, so the choice
is moot.
B) Another device is sharing, and it has already connected. This will
result in your device failing to connect its interrupt.
C) Another device is sharing, and it has not already connect. This will
result in your device working and the other driver will experience a
failure to connect an interrupt. (Please don't write any device drivers
that actually cause other devices to fail to function.)
2. I'm not sure exactly what you mean by this. But, again, because PCI
devices may be forced to share interrupts, I'd like to write for a
moment on the topic of affinity. If your device is sharing, then it
must share each and every processor with the other devices on the
chains. Consider this example. You're sharing with a SCSI controller.
The SCSI controller has already started, connecting its ISR to processor
A and processor B. The input on the interrupt controller will
subsequently be unmasked, and directed to both processors. Now your
device gets IRP_MN_START_DEVICE with an affinity mask that includes both
processors. You decide to change that mask to include only processor B,
and you connect your interrupt. This will cause the kernel to connect
your ISR to processor B's chain, but not to processor A's. Now your
device interrupts. It may go to either processor. If it goes to
processor B, everything works fine, since your ISR will run. If the
next interrupt goes to processor A, then processor A will start running
through its ISR chain. Since your driver's ISR is not in that chain,
this interrupt cannot be dismissed. (I consider it a bug in the PCI
spec that there is no way to dismiss an interrupt without running a
driver-supplied ISR. But that's another discussion, particularly
because we've gotten that fixed in PCI 2.3.) When processor A gets to
the end of the chain, it acks the interrupt, even though no ISR has
claimed it. Since PCI interrupts are level-triggered, this will cause
the interrupt to be immediately re-asserted. Most SMP machines are
built with APIC interrupt controllers that will re-assert this interrupt
on the same processor that just failed to handle it, causing processor A
to now go into an endless loop, failing to handle the interrupt.
Processor A will remain at the associated Device IRQL, never dropping
down low enough to handle DPCs. The machine will continue to run only
as long as it doesn't depend on processor A to handle a DPC.
Dependencies of this sort usually take between 30 and 90 seconds to crop
up. At that point, the machine will appear completely hung to the user.
The point I'm tring to make is that you should never do anything with
your affinity mask other than just pass it through.
3. This seems harmless, though it seems just as likely to reduce the
performance of the machine.
4. If you have that much control over your environment, this might be
the way to go. But, again, if you do this at high IRQL, you'll
eventually deadlock the machine. If you do it at low IRQL, you'll may
still have latency problems.
5. The DIRQL that you connect with is used mainly when you call
KeSynchronizeExecution, so that the system can take the spinlock at the
right IRQL. The DIRQL that your ISR is called at is determined by the
vector that you're attached to. You can't help that.
The latency between when a device interrupts and when its ISR is called
is mostly a matter of waiting for other ISRs at equal or greater IRQL.
Or it's a matter of waiting for code that has explicitly raised IRQL.
- Jake Oshins
(the guy who maintains interrupt-related stuff in the NT kernel)
-----Original Message-----
Subject: ISR Latency question
From: "Assaf Wodeslavsky" <xxxxx@hotmail.com>
Date: Tue, 25 Dec 2001 00:21:06 +0200
X-Message-Number: 1
if you want to reduce latency do this:
1. use exclusive vector when connecting
otherwize, the kernel will loop through all registered isr's (on
your
shared vector)
having each one query it's hardware
2. use affinity when connecting if you are smp
3. call your isr in the background!!!
to make sure that the TLB, code cache and data cache are not
flushing
your isr away
4. you might consider running on smp and not using interrupts at all
simply busy loop on one processor waiting for the int# condition
(osr's book hints at this)
5. you might want to connect as a higher DIRQL then the system assigns
you,
if you feel other devices are bothering you.
the bottom line is to understand, in my oprinion, that interrupt latency
is
not an issue of how long before the cpu gets interrupted!
that happens very quickly (and is 100% deterministic - you should be
able to
find real numbers from the pc manufacturer's data sheets),
but rather, the time it takes the cpu to translate the IDT virtual
address
into a physical address, and then to access variables from RAM, needing
to
translate each one's address from virtual to physical (requiring 3 RAM
accesses).
Also, if using shared irq's, the actual IDT's isr has to loop through
all
isr's on the same irq, each one needing to go through the same story.
So, in my opinion, it is not hardware interrupt latency that is
producing
the large delays we always hear about,
but rather Operating System issues, such as searching for the proper
isr,
TLB and the caches.
call me if you want to discuss this further
056-657-169
regards to Asher, Mimi, Yeoshua and the rest of the gang at excalibur
assaf
----- Original Message -----
From: "Avi Shmidman" <xxxxx@excalibur.co.il>
To: "NT Developers Interest List" <xxxxx@lists.osr.com>
Sent: Monday, December 24, 2001 3:05 PM
Subject: [ntdev] ISR Latency question
>
> During the "latency" between the occurence of an interrupt and the
time an
> ISR receives control, is the operating system actually preprocessing
the
> interrupt? For instance: if we say that we find an average latency of
50us
> until entering the ISR, does that mean that every interrupt generated
> steals 50us of time from the processor (plus the actual time in the
ISR,
> plus the post processing)? Or is this simply the time it takes for the
> processor to get around to processing the interrupt (due to higher
priority
> interrupts, perhaps)?
> Having set bpint breakpoints in softice and traced through, I am aware
that
> there is indeed quite a bit of system preprocessing code which is
executed
> upon every interrupt. Nevertheless, the latencies we talk about, in
the
> tens of microseconds, seem much larger than necessary to run that
> preprocessing code.
>
> Sincerely,
> Avi
>
>
> ---
> You are currently subscribed to ntdev as: xxxxx@hotmail.com
> To unsubscribe send a blank email to
leave-ntdev-$subst('Recip.MemberIDChar')@lists.osr.com
<...excess quoted lines suppressed...>
---
You are currently subscribed to ntdev as: $subst('Recip.EmailAddr')
To unsubscribe send a blank email to
leave-ntdev-$subst('Recip.MemberIDChar')@lists.osr.com
]]>
|
ISR Latency question