So, you’ve got a driver.  Maybe it’s a HID driver for the USB stack.  Or perhaps it’s a filter driver in the storage stack; or a class driver.  Whatever kind of driver it is, you have a dispatch entry point – let’s say for read IRPs – that looks something like the following:

Code Segment 1:

NTSTATUS

YourDispatchRead(PDEVICE_OBJECT DeviceObject,

              PIRP Irp) {

       PIO_STACK_LOCATION ioStack;

       PYOUR_DEV_EXT devExt;

       ioStack = IoGetCurrentIrpStackLocation(Irp);

       devExt = DeviceObject->DeviceExtension;

       //

       // you do some validation here

       //

       IoCopyCurrentIrpStackLocationToNext(Irp);

       //

       // Maybe you manipulate some of the IRP

       // stack parameters here.

       //

       return(IoCallDriver(devExt->LowerDriver, Irp));

}

There’s certainly nothing too remarkable about this dispatch routine.  You get the IRP, you set up the next I/O stack location and, you pass the IRP down to the driver below you.  The status you return is whatever status the driver below you, passes you back to you.

And this code works without any problem.  You’ve seen, perhaps even written, this code about a million times.

Though your driver’s working fine, you decide that for whatever reason, you want to keep track of the number of requests that you have in progress on the driver below you (the one to which you’re sending IRPs).  To do this, you modify the code in your dispatch routine slightly:

Code Segment 2:

      // (code prior to this unchanged)

       // you do some validation here

       //

       IoCopyCurrentIrpStackLocationToNext(Irp);

       //

       // Maybe you manipulate some of the IRP

       // stack parameters here.

       //

       //

       // Keep count of IRPs in progress

       //

       InterlockedIncremen(&devExt->IrpsInProgress);

       //

       // Set a completion routine

       // where we can decrement the count

       IoSetCompletionRoutine(Irp,

                     YourCompletionRoutine,

                     NULL,

                     TRUE,

                     TRUE,

                     TRUE);

       return(IoCallDriver(devExt->LowerDriver, Irp));

}

Before passing an IRP down the stack, you increment a counter.  You then set a completion routine in the IRP, and send the IRP to the lower driver.  You decrement the counter in the completion routine, as follows:

Code Segment 3:

YourCompletionRoutine(PDEVICE_OBJECT DeviceObject,

PIRP Irp,

PVOID  Context) {

       PYOUR_DEV_EXT devExt;

       InterlockedDecrement(&devExt->IrpsInProgress);

       return(STATUS_SUCCESS);   

}

Pretty tame stuff, right?  You only have one problem: Every once in a while, the system gets weird.  Applications talking to your driver sometimes “hang”.  And what’s really strange is that when this happens your counter in progress IRPs indicates no I/O requests are pending in driver’s below yours.  Driver Verifier (at least on XP) will also fail your driver in this case.  What’s going on?

Welcome to the strange world of completion routines!  It’s a world where sometimes things that seem like they shouldn’t work, function perfectly well. And where things that look to be perfect are in fact seriously broken.

In such a world, how can you be certain that your code will work the way you intend it to?  Well, one way is to fully understand and apply the details of how NT processes and completes I/O requests.  This is certainly do-able, but it’s a non-trivial undertaking.  OSR consulting partner Tony Mason explained the details of I/O completion in his landmark article in The NT Insider (May-June 1997, How NT Handles I/O Completion). We also teach the details in our Advanced Driver seminar.  Another, more pragmatic, method of ensuring your driver will work the way you intend is to follow a simple set of rules.  In this article we’ll concentrate on describing these rules, and on illustrating how they are applied.

Proper IRP Handling

If you’re reading this article, I’m hopin’ that you already know the basics of how to write a proper Dispatch Routine. But let’s briefly discuss dispatch routines anyway.  Bear with me.  What’s important about this is that it’s in the Dispatch Routine that we encounter our first two rules:

Rule 1: If you complete the IRP passed to your Dispatch Routine in your Dispatch Routine, you must (a) Fill the IRP’s completion status and information into the IRP’s I/O status block, (b) Call IoCompleteRequest(), and (c) return the status you used to complete the IRP (the status value you filled into the .Status member of the I/O status block in the IRP) as the return value from your Dispatch Routine.

Duh!  Kindergarten stuff, right?  Note that you have to do all three of these things, or your code will not work correctly.  If you want your driver to work, you can’t call IoCompleteRequest() without first having setup the I/O Status Block.  Similarly, you can’t return any status you want from your Dispatch Routine.  You have to return the status with which the IRP was completed.  Obviously. Following this rule, we’ve all written the equivalent of the following code about a million times:

      Irp->IoStatus.Status = STATUS_SUCCESS;

       Irp->IoStatus.Information = bytesTransfered;

       IoCompleteRequest(Irp, IO_NO_INCREMENT);

       return(STATUS_SUCCESS);

}

Rule 2: If you return from your Dispatch Routine without completing the IRP passed to you, you must (a) Mark the IRP pending, before the IRP is completed to your caller, by calling IoMarkIrpPending(), and (b) return STATUS_PENDING from your Dispatch Routine.

Again, no big deal.  But again, let’s just make sure to note that you have to do both of these things, or your code will not work properly.  You have to both return STATUS_PENDING and mark the IRP pending to properly pend the IRP.  And, obviously, you have to mark the IRP pending before any other function has a chance to grab the IRP, complete it back to the driver that called you and the I/O Manager. So, for example, you need to call IoMarkIrpPending() before you stick the IRP on your queue of pending IRPs (where it could be removed, processed, and completed before you marked it pending).  Thus, the right way to do this is:

IoMarkIrpPending(Irp);

       ExInterlockedInsertTailList(&devExt->ListEntry,

                     &Irp->Tail.Overlay.ListEntry,

&devExt->ListSpinLock );

       return(STATUS_PENDING);

}

There’s an important subtlety to understand about Rule 2.  Note that it says you must mark the IRP pending “before the IRP is completed to your caller”.  This shouldn’t be interpreted to mean you must to mark the IRP pending before you return from your Dispatch Routine.  After all, no driver (including the driver above you) has access to an IRP from its Dispatch Routine after returning from IoCallDriver().  So, as long as you mark the IRP pending before the IRP gets completed, everything will be fine.

One of the XP developers talks about the two basic rules presented above as being the contract that your driver enters into with the driver above you. That’s not a bad way to think about it.  And, believe it or not, most of the problems that people have with I/O completion, including completion routines, come down to properly sticking to the contract – that is, scrupulously obeying Rule 1 and Rule 2, above.

Pending Problems

So, what do Rule 1 and Rule 2 tell us about why the driver in our previous example failed?  If you apply the rules carefully, it should become clear what the problem is.  In Code Segment 2, you just did:

       return(IoCallDriver(devExt->LowerDriver, Irp));

}

Problems occur when the driver below you returns STATUS_PENDING.  When it does, you will also return STATUS_PENDING, right?  And as a result you will break the contract you have with the driver above you that says, according to Rule 2, that if you return STATUS_PENDING you must also mark the IRP pending!

So, how do you fix this?  Just honor the contract.  Let’s hope you don’t try to fix it this way:

Code Segment 4 (broken code):

          status = IoCallDriver(devExt->LowerDriver, Irp);

       if (status == STATUS_PENDING) {

              IoMarkIrpPending(Irp);

       }

       return(status);

}

Well, if you do try this, you won’t be the first and you won’t be the last.  Do you see the problem in this code?  As soon as you call IoCallDriver(), your Dispatch Routine no longer has access to the IRP.  The IRP could already be completed, and even freed back to pool by the I/O Manager, by the time you return from IoCallDriver().  So, if you do call IoMarkIrpPending() in the above fragment, you could be scribbling on pool.  Good luck finding that one when the system bugchecks calling some other driver’s timer DPC, or whatever.

So, what’s the secret to fixing code segments 2 and 3, above?  The question you need to answer is: When do you have access to the IRP (so you can mark it pending) after you’ve called IoCallDriver(),  but before the IRP is completed back to the driver above you?  The answer is: In your completion routine.

Completion Routine Madness

You fix the IRP pending problem by checking in your completion routine to see if the driver immediately below you has returned STATUS_PENDING (which will cause you to return STATUS_PENDING) and if it has, by calling IoMarkIrpPending().  This allows you to fulfill your contract with the driver above you. Fortunately, the I/O Manager tells you in your completion routine if the driver below you returned STATUS_PENDING by setting the PendingReturned flag in the IRP to TRUE.  Thus, if Irp->PendingReturned is set in the IRP in your completion routine, you call IoMarkIrpPending().  The code is as follows:

Code Segment 5:

YourNewCompletionRoutine(PDEVICE_OBJECT DeviceObject,

PIRP Irp,

PVOID  Context) {

       PYOUR_DEV_EXT devExt;

       InterlockedDecrement(&devExt->IrpsInProgress);

       If(Irp->PendingReturned) {

              IoMarkIrpPending(Irp);

       }

return(STATUS_SUCCESS);   

}

Not hard, right?  But this does mean we have a new rule:

Rule 3: If you return the status from the driver below yours as the status from your dispatch routine, the IRP’s pending status, as indicated by Irp->PendingReturned, must be propagated with a completion routine.

But Wait, There’s More!

By now, you must be wondering about the original version of the driver, the version that worked, shown in code segment 1.  That driver just did return(IoCallDriver(…)).  Why did that driver work?  Doesn’t that driver also break Rule 3, and break the contract inherent in Rule 2 when the underlying driver returns STATUS_PENDING?  Well, you need our last rule, Rule 4, to explain that:

Rule 4: If you do not provide a completion routine for an IRP, the I/O Manager will provide one for you that properly implements Rule 3.  If you do provide a completion routine for an IRP, the I/O Manager assumes you will properly implement Rule 3.

In an effort to “help you out” the I/O Manager provides a default completion routine.  If he didn’t do this, every driver that passed an IRP to a lower driver would have to implement a completion routine.  And what’s the point of every such driver in the system including the exact same code?  None.  So, the I/O Manager provides you a default routine that “does the right thing.”

Of course the problem with this type of help is that many driver writers don’t know they’re getting it.  And when the time comes to provide their own completion routines, they don’t know that they are now responsible for propagating the IRPs pending status.

You might ask, “Why doesn’t the I/O Manager just always propagate the IRP’s pending status and save us all this hassle?”  The answer is that if you do not have completion routine, there is no way for you to (validly) alter the status with which an IRP is completed by a driver below you.  When you implement a completion routine, you re-gain ownership of the IRP – and therefore become responsible for the status with which the IRP is completed.  You could, for example, choose to block in your dispatch routine, and not return to your caller the status that was returned to you from IoCallDriver().  There are too many variations possible, and so (as is usually the case) NT lets you handle it so as not to constrain you.

Finally, note that it’s considered poor engineering practice to propagate the pending status of an IRP when this isn’t necessary.  Given the confusion surrounding this topic, always calling IoMarkIrpPending() when you find Irp->PendingReturned set to TRUE in a completion routine can lead to misunderstanding the code that’s been implemented.  So, for example, you don’t want to propagate the IRPs pending status when you return STATUS_MORE_ PROCESSING_REQUIRED from your completion routine, such as when you reclaim an IRP_MN_START_DEVICE in PnP processing. Worse then just a misunderstanding, this can actually cause subtle, timing-sensitive, code bugs. Consider the following code from the DDK’s HID Game driver (DbgPrint type statements removed for clarity):

NTSTATUS INTERNAL

    HGM_PnpComplete

    (

    IN PDEVICE_OBJECT   DeviceObject,

    IN PIRP             Irp,

    IN PVOID            Context

    )

{

    NTSTATUS ntStatus = STATUS_MORE_PROCESSING_REQUIRED;

    PAGED_CODE();

    UNREFERENCED_PARAMETER (DeviceObject);

    KeSetEvent ((PKEVENT) Context, 0, FALSE);

    /*

     *  If the lower driver returned PENDING, mark our stack location as

     *  pending also. This prevents the IRP's thread from being freed if

     *  the client's call returns pending.

     */

    if(Irp->PendingReturned)

    {

        IoMarkIrpPending(Irp);

    }

    return ntStatus;

}

The bug in this PnP completion routine was reported as part of the Windows DDK Bug Bash.  See the problem?  OK, actually, there’s more than one.  But the BIG problem?  The code sets event, a pointer to which is passed in as the Context value.  As is typical for PnP processing, the main-line code (dispatch PnP routine) is waiting on this event so when it’s set, it resumes processing the power request that has been completed by the underlying bus driver, and then calls IoCompleteRequest().  The problem comes in the completion routine when driver then goes on to (unnecessarily) check the pending bit and call IoMarkIrpPending() if the underlying bus driver returned STATUS_PENDING.  The only problem is, on an SMP system, by the time this check is performed the IRP could already have been completed by the main-line code that was awoken by the KeSetEvent(). 

There’s plenty else to dislike about this particular completion routine. Like the fact that the completion routine is marked as PAGED_CODE().  Huh?  Can’t completion routines be called at IRQL DISPATCH_LEVEL.  Never mind, this isn’t a code review, and hence I digress.

Pending Perfectly

So, now you’ve seen the rules.  If you follow them, you will never again be bitten by problems to do with returning STATUS_PENDING in your dispatch routine and using a completion routine.  Oh, there are a lot more subtleties to do with the use of completion routines.  We’ll have to talk about those another time.  Pend well!