IoSetCompletionRoutine() is a macro and its definition is in ntddk.h (See Figure 2). 

Figure 2

One common instance where a completion routine is used is when a driver creates and manages its own IRP pool. In this case, the I/O completion routine traps the IRP, returns it to the driver’s private IRP pool, and then returns STATUS_MORE_PROCESSING _REQUIRED to the I/O Manager. This causes the I/O Manager to immediately cease processing the IRP completion and leaves the IRP with the driver.

The important point to notice about I/O completion routines, looking at the Figure 1, is that your driver’s completion routine is not in your I/O stack location, but in the I/O stack location of the next driver. Why is it set up this way? One reason is that the last driver called does not need an I/O completion routine; After all, it is ultimately the driver performing the I/O completion (by calling IoCompleteRequest()). There is therefore no need for the I/O Manager to notify the lowest level driver that it just completed the I/O.

Another reason a driver’s I/O completion routine is in the next I/O stack location concerns the way the IRP is initially built. The kernel mode component, such as a driver, that initially allocates the IRP does not require an I/O stack location. However, it is possible that it might require an I/O completion routine (to allow it to return the IRP to its private pool, for example). Recall that the last driver in the calling chain does not require a completion routine but does require an I/O stack location. Thus, for space efficiency, the N-1^st driver’s completion routine is stored in the N^th driver’s I/O stack location.

Since the I/O Manager does not use I/O completion routines itself, if you are developing a highest level driver you will almost never see a completion routine in your I/O stack location. Some system components, however, create their own IRPs and pass them along to highest level drivers, specifying their own I/O completion routines. For example, SRV (the Lan Manager File Server) and NTVDM (the MS-DOS emulation layer) both build their own IRPs and pass them to highest-level drivers. And these components both set I/O completion routines into the IRPs they create.

As a result of misunderstanding the location of completion routines in the I/O stack, a very common mistake occurs when passing an IRP from your driver to an underlying driver. This mistake, illustrated in Figure 3, is to simply copy the current I/O stack location to the next driver’s I/O stack location. The problem is that copying the contents of the current stack location to the next stack location also copies the I/O completion routine along with it. This works fine, as long as there is no completion routine in the current stack location. If a completion routine was supplied, it will now appear in two different I/O stack locations, with the obvious result of it being called twice. This is generally not good, and results in an eventual blue screen or other unstable system behavior. Of course, the caller’s code could be written to handle this case and everything will still work fine. But few driver writers anticipate that their completion routines will be called incorrectly!

PIO_STACK_LOCATION IrpSp;
PIO_STACK_LOCATION NextIrpSp;

IrpSp = IoGetCurrentIrpStackLocation(Irp);
NextIrpSp = IoGetNextIrpStackLocation(Irp);

*NextIrpSp = *IrpSp;

return IoCallDriver(NextDeviceObject, Irp);

Figure 3

For file system filter drivers, this problem often first surfaces when you begin filtering drives being served by SRV. SRV provides its own I/O completion routines and does not protect against having its completion routine called with a different driver’s I/O stack location. The fix for this is to be sure that you always call IoSetCompletionRoutine() after copying the I/O stack location in your driver.

Of course, even if you write your filter driver code correctly, there might still be problems. Once example of the problems you can see is present in NTFS in NT 4.0. As late as SP2, NTFS did not clear the I/O completion routine when copying the I/O stack during IRP_MJ_DEVICE_CONTROL function processing. Thus, there were circumstances under which our completion routine was being called twice (once with NTFS’ device object!).

The defense against this particular problem is straight-forward. In your device object’s extension area, you should keep some identifying signature: A unique value that you can use to determine (with high probability, at least) that this is your device object and not some other driver’s device object being passed to your completion routine. If your completion routine is called with a device object that’s not yours, you simply need to emulate the logic the I/O Manager would use had there been no completion routine. See the code fragment in Figure 4 that demonstrates this. Now, if your completion routine is called with some other driver’s device object, everything will work correctly.

Figure 4

A less common, but equally daunting problem we found recently was that prior to NT 4.0 Service Pack 2, there were certain cases in CDFS for IRP_MJ_CLOSE where the IRP was not being properly completed. Because of the mechanics of the I/O Manager, things seem to work correctly, even with the checked build (more on why this is the case in a moment). For a filter driver, however, the symptom of the problem is that the filter driver’s completion routine is never called. While this was fixed in Service Pack 2, for those people who must support their product on earlier NT 4.0 versions or wish to handle this problem correctly should it reappear in the future, there is a work-around that relies upon the I/O completion processing. The code fragment in Figure 5 describes a mechanism for detecting and working around this problem. This mainline code is then combined with a completion routine (Figure 6) which sets the event.

Figure 5 

Figure 6

The problem was that CDFS was returning STATUS_SUCCESS without ever calling IoCompleteRequest(). Indeed, it turns out that for most I/O operations, if a driver returns anything except STATUS_PENDING in its dispatch routine, the IRP being dispatched will be completed by the I/O manager. Of course, you should never write your code to do this. Completing IRPs without calling IoCompleteRequest() is a logic error in the driver that does it, and can cause all sorts of nasty problems in higher level drivers.

IRPs get completed in the case where anything except STATUS_PENDING is returned because the lower level driver should have already called IoCompleteRequest(), which in turn would have called your driver’s I/O completion routine. This leads us to the most complicated part of I/O completion – understanding how the I/O Manager actually does I/O completion.

Role of The I/O Manager

It turns out that the I/O Manager implements I/O completion processing in two separate and distinct stages. The first stage is thread context-independent, meaning that it can be performed in any arbitrary thread context. The second stage is thread context-dependent – this means the step must be performed in the context of a specific thread, in our case the context of the thread that originally requested the I/O.

In the first stage of I/O completion processing, the IRP is "unrolled" and the I/O completion routines of each driver are notified that the I/O operation has completed. During this stage of processing, any driver that provided an I/O completion routine will be called back and will have a second opportunity to look at the IRP.

In the second stage of I/O completion, information from the IRP is copied back into the thread’s user-space memory buffers. Thus, this operation must be done in the correct process context. Of course, even for calls where the bulk of data need not be copied, such as in the case for direct I/O, there is always some information that needs to be copied, such as the I/O status block. Once this is done, the IRP itself is torn down, which includes discarding any MDLs associated with the IRP and finally the memory of the IRP itself is freed.

The first stage of I/O completion is done when a driver calls IoCompleteRequest(). Note that IoCompleteRequest() is a void function, so the caller is provided with no additional information about the I/O completion. Indeed, the caller does not even know if the first stage of I/O completion is done, as a higher-level driver might have returned STATUS_MORE_PROCESSING_REQUIRED which suspended further first stage I/O completion handling.

The second stage of I/O completion happens at an arbitrary time after IoCompleteRequest() is called. This could mean that second stage I/O completion is done by the time IoCompleteRequest() returns, or it could even happen at some later time, depending upon other activities ongoing in the system.

The actual ordering of events is based upon the status of the IRP after the last I/O completion routine has been called. At that point, the I/O manager looks at the IRP to determine if STATUS_PENDING was returned (or will be returned since we don’t know the exact order of operations at this point) from the highest level driver. If STATUS_PENDING was returned from the highest level driver, then the I/O manager queues an Asynchronous Procedure Call (APC) to perform the secondary I/O completion. It performs this check by looking at the PendingReturned field within the IRP itself.

Here’s where it gets interesting. If STATUS_PENDING was not returned from the highest level driver, then the I/O manager simply returns control to the caller (i.e., it returns back to the driver that made the IoCompleteRequest() call). Recall that at this stage, the driver that called IoCompleteRequest() has no knowledge of the state of the IRP.

So where does the second stage I/O completion occur in this case? Why, in the I/O Manager, of course! In Figure 7 we provide a graphical description of the flow of control in this case (the "synchronous I/O case").

Figure 7

Before analyzing I/O completion further, it is a good idea to review each of these individual steps:

1. The application program issues an I/O request (read, write, etc.) to the I/O Manager.
2. The I/O Manager dispatches the I/O request by building an IRP and passing it to the first driver in the chain.
3. The first driver continues processing by sending the I/O request to the second driver.
4. The second driver continues processing by sending the I/O request to the third driver.
5. The third driver completes the I/O operation. It calls IoCompleteRequest() and initiates first stage completion.
6. During first stage completion the second driver’s completion routine (if any) gets called back.
7. During first stage completion the first driver’s completion routine (if any) gets called back.
8. Next, IoCompleteRequest() returns control back to the third driver. First stage I/O completion is done.
9. The third driver returns a status value (e.g., SUCCESS) to the second driver.
10. The second driver returns a status value to the first driver.
11. The first driver returns a status value to the I/O Manager
12. The I/O Manager notes that second stage processing must be performed and initiates second stage I/O completion.
13. The second stage I/O completion routine returns the status of the I/O operation and copies any data out to the application’s address space. Once that is done, the IRP is dismantled and discarded.
14. The second stage I/O completion routine returns control to the I/O Manager.
15. The I/O operation is now complete and control returns to the caller.

The interesting point in this figure is step 12 where the I/O Manager initiates the second stage I/O completion processing. This works correctly because the call completed synchronously and the I/O Manager "knows" that it is in the correct thread context AND the first stage I/O completion processing has been properly completed.

Recall the CDFS problem in NT 4.0 we described earlier, where in one particular situation CDFS failed to call IoCompleteRequest(), and instead just returned STATUS_SUCCESS in its dispatch function? Well, the reason that it worked correctly was because the I/O Manager was performing the stage two completion processing, thus making sure that the information was being transferred back to the application program as well as freeing the IRP. Of course, any intermediate drivers never learned about that I/O completion. Stage one processing (steps 5 through 8 in the diagram) was never performed!

In the asynchronous case, these events occur in a very different order. This is shown in Figure 8.

Figure 8

Again, here is a description of each of the labeled steps:

1. The application program issues an I/O request (read, write, etc.) to the I/O Manager.
2. The I/O Manager dispatches the I/O request by building an IRP and passing it to the first driver in the chain.
3. The first driver continues processing by sending the I/O request to the second driver.
4. The second driver continues processing by sending the I/O request to the third driver.
5. The third driver completes the I/O operation. It calls IoCompleteRequest() and initiates first stage completion.
6. During first stage completion the second driver’s completion routine (if any) gets called back.
7. During first stage completion the first driver’s completion routine (if any) gets called back.
8. Next, IoCompleteRequest() queues the I/O completion APC to handle second stage I/O completion. First stage I/O completion is done.
9. The I/O completion APC is dequeued and second stage I/O completion starts. Note that this is done in the context of the thread which originated the I/O.
10. The I/O status and data is copied to the user address space.
11. Next IoCompleteRequest() returns back to the third driver.
12. The third driver returns a status value (e.g., SUCCESS or PENDING) to the second driver.
13. The second driver returns a status value to the first driver.
14. The first driver returns a status value to the I/O Manager.
15. The I/O operation is now complete and control returns to the caller.

In this case (the "asynchronous I/O" case) the second stage I/O completion processing (steps (9) and (10)) occur in an arbitrary order relative to the return of control from IoCompleteRequest() back to the application program (steps (11) through (14)).

The actual return to the application (step (15)) may be serialized by the I/O Manager, such as in the case where the original calling application asked for synchronous I/O behavior. Of course, the I/O Manager does this the same way any other Windows NT kernel mode code would do it – by waiting on an event (in this case, an event in the IRP). When the stage two completion processing is done, that event will be set and the status of the I/O operation is returned to the calling application (step (15)).

Also note that the steps of calling from driver to driver, and the subsequent return, could even each be performed asynchronously. Thus, if the first driver in the chain wishes to queue the request and process it asynchronously, it would return STATUS_PENDING to the I/O Manager (step (14)).

If the caller did not request synchronous behavior, then at step (15) the I/O Manager returns control back to the caller, indicating that the I/O is in progress. During stage two I/O completion processing the I/O is then completed and the status of the operation is copied back into the appropriate address space. The application can then determine that the I/O has completed by using one of the existing mechanisms, such as an APC routine, waiting on an event, or by polling to see if the I/O is done. In the interim, the thread that initiated the I/O can continue handling normal processing, or even start additional I/O operations.

Setting Up Your Completion Routine

Because the use of completion routines within certain types of drivers is fairly common, understanding the model used within Windows NT for I/O completion processing can help developers build more robust drivers and locate problems within their own driver. One common example of this is code we have seen used numerous times in an attempt to trap IRP_MJ_CREATE calls above a file system. The new filter driver writer will attempt to write something like Figure 9, and in their completion routine do something like Figure 10.