You're writing a device driver. It might be a driver for a device that uses Programmed I/O (PIO) or a device that supports Direct Memory Access (DMA). In either case your device interrupts to let your driver know when a pending request has been completed.

To keep the system as responsive as possible to interrupts, a driver's interrupt service routine (ISR) is typically restricted to performing only the minimum, most time critical, processing. If any additional processing is required beyond what can be done in the ISR, the driver will queue a Deferred Procedure Call (DPC) for ISR completion (DpcForIsr). This DpcForIsr runs at IRQL DISPATCH_ LEVEL. Within its DpcForISR the driver performs whatever additional processing is required to complete its request.

Because a well-written driver is informed of an event in its ISR, but does most of its event processing within its DpcForIsr, a challenge many new driver writers face is how to pass event information from the ISR to the DPC. This article describes a few potential solutions to this problem.

One Request at a Time

Not every driver has a difficult time communicating event information from its ISR to its DPC. For example, if your driver only supports having one request in progress at a time, passing information from your ISR to the DPC doesn't present much of a challenge. This is because Windows provides a simple, built-in mechanism for you to use that's part of the IoRequestDpc function.

There are several ways a driver can queue a request to have its DpcForIsr function executed.  The most common way a driver does this is by calling the function IoRequestDpc As a review, the prototype for this function is as follows:

VOID

IoRequestDpc(IN PDEVICE_OBJECT DeviceObject,

IN PIRP Irp,

IN PVOID Context);

Notice that the first parameter passed to IoRequestDpc is a pointer to your driver's Device Object. The I/O Manager uses this pointer to identify the DPC Object that is embedded in the Device Object. This DPC Object is used to queue the DPC request. The Irp and Context parameters can be used to pass any information you want from the ISR to the DpcForIsr. Yes, that's right. Even though the Irp parameter is typed as PIRP, the I/O Manager never references this parameter except to pass it to the DpcForIsr. So there's no reason why this parameter must be a pointer to an IRP.

As a result of your driver's call to IoRequestDpc, the I/O Manager will queue a callback to your driver's DpcForIsr. The prototype for your driver's DpcForIsr is:

VOID

DpcForIsr(IN PKDPC Dpc,

IN PDEVICE_OBJECT DeviceObject,

IN PIRP Irp,

IN PVOID Context);

Assuming the driver processes only one request at a time per device, and that the device only generates one interrupt per request, the DeviceObject, Irp, and Context parameters are the values that were provided when the driver called IoRequestDpc. In this case, getting data from your ISR to the DPC isn't difficult at all!

The Problem

So, you may be wondering when and why is passing information from your ISR to your DpcForIsr problematic. Consider how IoRequestDpc passes the Irp and Context parameters you provided to your driver's DpcForIsr and how DPCs are processed by the system.  Assuming the DPC Object for the device you identified is not already in use (and assuming you have not explicitly changed the Importance or TargetProcessor attributes of the DPC object) IoRequestDpc completes the following tasks:

• Copies the parameters you provided to the DPC Object embedded in the Device Object to which you referred the call.
• Inserts the DPC object at the end of the current processor's DPC queue.
• Generates an IRQL DISPATCH_LEVEL software interrupt. 

As a result, the next time the system is ready to return to an IRQL less than IRQL DISPATCH_LEVEL, it checks the DPC queue on the current processor. If the DPC queue is not empty, the system removes the DPC Object at the head of the queue, and calls the DPC callback function indicated in the DPC Object, passing the parameters contained in the DPC Object. If your driver queues a DPC request by calling IoRequestDpc, the callback function will be your driver's DpcForIsr and the parameters passed will be those you specified when IoRequestDpc was called.

The problem arises when your driver calls IoRequestDpc and its DPC Object is already in use. In other words, the DPC Object embedded in your driver's Device Object is already on one of the system's DPC queues. In this case, IoRequestDpc simply returns, and the values you provided as the Irp and Context parameters are effectively ignored. In case you're wondering, IoRequestDpc doesn't return any value, so there's no way to determine whether the parameters you provided to your DpcForIsr.

To summarize this problem ‑ if there's any chance your driver will call IoRequestDpc while the DPC Object associated with the Device Object is already queued, you can not rely on IoRequestDpc to pass information from the ISR to the DpcForIsr. This means the only time you can use IoRequestDpc to communicate information from the ISR to the DPC is if the following two requirements have been met.

1.      The driver supports only one active I/O request at a time per device.

2.      The device will never interrupt more than once per request without the DpcForIsr running between each invocation.

Fortunately these requirements are easily met by drivers for most simple devices. Even a driver for a traditional IDE disk could use IoRequestDpc without any problem. However, as time goes by and devices become faster and smarter, there are an increasing number of devices that support multiple simultaneous I/O requests. The drivers for these devices need to use alternative mechanisms for communicating context information from their ISR to their DPCs.

Solution One: You Get Lucky

In some cases, you might just "luck out" and your device hardware will solve the problem for you. Consider the case of a device that supports one read and one write operation simultaneously. This device might have a status register layout something like the following:

typedef struct _DEV_STATUS_REG {

   ULONG  InterruptRequested:1;

   ULONG  InterruptAck:1;

   ULONG  ReadComplete:1;

   ULONG  WriteComplete:1;

   ULONG  ReservedZero:4;

   ULONG  ReadStatus:8;

   ULONG  WriteStatus:8;

   ULONG  SomeOtherStuff:8

} DEV_STATUS_REG, * PDEV_STATUS_REG;

This illustrates a device status register that has separate completion and status bits for each type of request that can be in progress simultaneously. Thus, in this example register layout, when a read operation is complete, the device will:

• put the read operation completion status in the ReadStatus field
• set the ReadComplete bit
• set InterruptRequested
• generate an interrupt

To service the interrupt, the driver's ISR will check the InterruptRequested bit to see if the device is interrupting. Assuming it is, the driver will acknowledge the interrupt and re-enable interrupts on the device. The driver then calls IoRequestDpc passing either no useful information or information that does not change from call to call for a given device in the Irp or Context parameters. 

Once the driver is in its DpcForIsr, it interrogates the device status register to determine the event or events that need to be serviced. If the ReadComplete bit is set in the device status register, the driver reads the operations completion status from the ReadStatus bits, and processes the request accordingly. Before leaving its DpcForIsr, the driver similarly checks the WriteComplete bit in the device status register to see if it has a write operation to complete.

Note however, that merely having the idealized register arrangement shown in this example is not sufficient for you to avoid ISR to DPC communication problems. To avoid these communication problems, your hardware must provide:

• separate read and write completion bits;
• separate mechanisms for determining read completion status and write completion status;
• bits that stay latched after the interrupt has been acknowledged so the driver can check them within its DpcForIsr. 

Device status bits that stay latched until they are cleared are often referred to as "write 1 to clear" bit. This means the driver is responsible for writing a bit value of 1 in the position of the bit being cleared. Thus, the driver code to clear the ReadComplete bit looks something like the following:

DEV_STATUS_REG clearRead;

clerRead.ReadComplete = 1;

WRITE_PORT_ULONG(devExt->StatusPort, clearRead);

Unfortunately it is rare to find hardware designs that are this considerate of Windows driver writers' needs. Instead, ReadComplete bits are typically cleared as soon as the interrupt is acknowledged by the driver even when separate status bits are provided.  Hardware designers probably think they're doing us a favor by automatically clearing these bits to help get ready for the next request. Sigh!

Solution Two: You Get Lucky ‑ But Only Partly Lucky

Let's assume your device has the same status register layout as described previously, but the ReadComplete bit, WriteComplete bit, and the associated status bits are automatically cleared as soon as the driver acknowledges the interrupt. Keep in mind that your driver can't use the parameters to IoRequestDpc to pass information from the ISR to the DPC. The good news is that there is a relatively simple solution to this problem.

The solution involves the driver keeping a private, duplicate copy of the device's status register. This copy of the register is often referred to as a "shadow" copy because it doesn't necessarily reflect the exact contents of the device'register at any point in time. Your driver keeps this shadow copy in non-paged space, typically in its Device Extension. 

In its ISR, the driver updates the shadow register by OR'ing the bits from the device's status register into the shadow copy. The driver must do this before acknowledging the interrupt to the device, which would result in the status bits being cleared. The code in Figure 1 illustrates the actions a driver takes in its ISR to maintain a shadow register copy.

Figure 1 - Maintaining a shadow register copy

In this interrupt service routine example you can see that the driver first makes a copy of the device's current interrupt status register. Then the driver OR's the status register copy into the shadow register that is maintained in the driver's device extension at devExt->ShadowStatus.

Within its DpcForIsr, the driver interrogates the status bits in the shadow register to determine the processing that it needs to perform. When the driver completes its DPC processing for one type of request, it clears the corresponding bits in the shadow register while holding the interrupt spin lock. The interrupt spin lock is required to ensure that the driver's access to the shadow register is serialized:

a) among all copies of its DPC that may be executing

            b) among all copies of its DPC and the driver's ISR

The example in Figure 2 illustrates this procedure.

Figure 2 - Processing Within a DPCForIsr

In this example code segment, which was taken from a driver's DpcForIsr, the driver begins by acquiring the interrupt service routine spin lock. This ensures the driver has exclusive access to the shadow register. In addition, it ensures that any updates the driver performs to the shadow register are performed atomically. With the interrupt spin lock held, the driver checks to see if the read complete bit is set. If it is, the driver stores the read completion status stored in the shadow register, and clears the ReadComplete bit in the shadow register. The driver then drops the interrupt spin lock, and checks to see if it needs to process the read complete. Later in its DpcForIsr, the driver will perform similar processing for write operations.

Note that in this example the processing performed with the interrupt spin lock held is kept to a minimum. This is because code executing with the interrupt spin lock held runs at device IRQL, which means it can block hardware interrupts at or below the device's DIRQL. Also note that the driver very deliberately does not attempt to "optimize" the time it spends holding the interrupt spin lock by checking both the read and write status bits.. Each time it interrogates the shadow register, the driver checks only to see if a single operation has completed. This facilitates maximum device throughput by taking advantage of the possibility that another instance of the device's DpcForIsr could be executing in parallel on a different processor.

Solution Two and One-Half: You Improvise

Of course, your device probably won't have a register layout as convenient as the one shown in the example. However, you should be able to expand the design used in the example and tailor it to meet your needs.

For example, let's say that the read status and write status on your device are stored in separate registers. Let's also assume these registers are separate from the one that holds the overall interrupt status. In this case, you might alter your design to use the following three shadow registers instead of one:

1.       Overall status register

2.       Read completion status register

3.       Write completion status register

If you use this approach, remember to protect all three registers with the interrupt spin lock!

Solution Three: You Are On Your Own

As you discovered from the previously described examples, as long as your device has separate status bits for each simultaneously active operation and some way to determine their statuses, passing information about I/O completion events from the ISR to the DPC is pretty straightforward. However, what should you do if this isn't the case? How do you handle a device that has some arbitrary amount of information that needs to be passed from its ISR to the DPC?

At OSR one of our favorite techniques for passing arbitrary information from the ISR to DPC is using ExInterlockedRemoveHeadList and ExInterlockedInsert TailList. The DDK documentation for these functions is, to say the least, horrendous. These functions take a pointer to a list head and a pointer to a spin lock.

(ASIDE: It's not like we haven't filed bugs on the docs for these functions, or that the documentation for these functions hasn't changed several times over the past few years.  We have, and it has.  Now, the documentation is merely impossible to use, as opposed to how it was before, which was confusing and subtly incorrect.  But I digress.)

What makes the interlocked list functions particularly well suited to the task of passing data from the ISR to the DPC is that they will work at any IRQL. This means these functions can be used to serialize access to a set of queues that are shared by the ISR and DPC without first having to acquire the interrupt spin lock. The trick to using the interlocked list functions correctly is that if you use an interlocked function to access a list, all accesses to that list must use these functions. You may never use an interlocked list function and a non-interlocked list function, such as the typical function RemoveHeadList, on the same list.

One method of communicating information from your ISR to your DPC using these functions is by maintaining a pair of lists, specifically a "source list" and a "completion" list. Your driver pre-fills the source list with a pool of packets (the format of which is defined by your driver).  To pass information from the ISR to the DPC, your driver removes a packet from the source list using ExInterlockedRemove HeadList. It then fills the removed packet with whatever information needs to be communicated to the DPC, and puts the packet onto the completion list using ExInterlockedInsertTailList.

In its DpcForIsr, your driver removes a packet from the completion list) and does the required I/O completion  processing. Once again, when completing this step, be careful to use ExInterlockedRemoveHeadList. When it has completed processing the I/O request represented by the packet, the driver returns the packet to the source list for re-use.

As an example of how a driver might use this scheme, consider a disk controller that can have a large number of I/O requests outstanding simultaneously. When a particular I/O request is passed to the controller, it is identified by a tag. When the request is completed, the controller supplies this tag, along with the ultimate status of the request, to the driver's interrupt service routine. The challenging part of designing a driver for this device is that multiple longwords of status information have to be communicated back to the DPC from the ISR, including completion status, byte count, and the tag value for the request. The driver uses the tag value to identify the IRP that corresponds to the request that was completed by the device.

The ISR code to support this example device might look something like Figure 3.

Figure 3 - ISR Code Sample

In this example code fragment from the driver's ISR, the driver checks to see if its device is interrupting. If it is not interrupting, it returns FALSE. If the device is interrupting, the driver removes a packet from the source list using ExInterlockedRemoveHeadList. It then fills this packet with the information needed to pass back to its DpcForIsr, and inserts the packet on the completion list using ExInterlockedInsertTailList. The example code finishes by requesting the driver's DpcForIsr by calling IoRequestDpc.

There is one error situation that driver's utilizing this dual-queue and packet scheme will need to handle - source queue depletion. In the example above, if the driver attempts to dequeue a packet from the source queue, but it finds the source queue empty, the driver bug checks. To avoid this, the driver could use one of the following two possible strategies:

• Put enough entries on the source queue so it will never run into an out of packet situation in its ISR.
• Keep count of the packets on the source queue, and stall the device if the source queue becomes depleted.

How your driver handles the source queue depletion problem is totally device dependent. The important point is that you must design a solution to this problem. Hopefully your solution will be something other than simply calling KeBugCheck!

The driver's DpcForIsr would probably utilize code that looks something like Figure 4.

Figure 4 - DpcForIsr Code Sample

In the example code fragment from a driver's DpcForIsr, the driver removes a packet from the completion list. If no packet is available, then there are no further completion requests outstanding and the DpcForIsr exits. Otherwise, if a packet was removed from the completion list, the driver processes it appropriately. When processing of the packet (and, presumably, the IRP associated with the I/O request) is completed, the driver replenishes the source queue by inserting the completed packet at its end. An important thing to notice about the DpcForIsr example is that the driver will continue to dequeue packets from the completion list until there are no further packets present on the list. This is absolutely necessary because, as discussed previously, several calls to IoRequestDpc can result in a single invocation of the driver's DpcForIsr. Because each remove operation is atomic, the driver's accesses to the completion queue are protected. Further, because each access to the list involves the removal of only a single packet, the possibility of parallel operation of DPCs on multiple machines is maintained. Finally, it's important to note that the code fragment shown above properly handles the case of the DpcForIsr being invoked when there are no packets on the completion queue. This situation could occur if the driver's ISR is invoked while the driver is simultaneously executing in its DpcForIsr, processing one or more previously completed requests from the same device. Don't laugh - It happens!

Not So Hard

Passing information from your ISR to your DpcForIsr can be complicated because it requires knowledge of many different Windows-related things: Serialization and locking, IRQLs, and the variety of potential DDIs that can be used at various IRQLs to name just a few. Like most things in your driver, designing an efficient and effective ISR to DPC communication strategy relies on a solid knowledge of your hardware and how it operates.

Hopefully, the ideas presented in this article will get you started in the right direction as you design a solution that best fits your driver's needs.