The Background

We periodically work with customers who are experiencing problems with drivers that they are developing.  Generally, we recommend a comprehensive review of the design and the code--a time-consuming process, because it starts off with a thorough code reading.  From that review we extract information about data structures and their interaction patterns.  Once we've constructed a "big picture" model (the design of the as-implemented code) we go back through the original source code to ensure that the implementation fits into this model.  This technique allows us to identify low level coding errors (e.g., "you are modifying this field without holding the appropriate lock") as well as higher level logical errors (e.g., "you aren't handling this range of conditions"). 

After reviewing many filters, the general "holes" are found by looking at the more esoteric cases that must be handled properly--even though they seldom happen on the developer's test machines or in the test lab, unless specific tests have been established to create these circumstances.

In this particular case, however, the client was experiencing a series of mysterious bug checks.  The most pronounced of these would manifest when the cache manager would crash accessing the file object in the shared cache map structure.

Here, the file object in use by the Cache Manager was different than the file object in use by the Memory Manager (see Figure 1).  Although unusual, this condition does happen.  We've never seen this cause a problem in the past, but in this instance what we observed is that the file object referenced by the Shared Cache Map did not actually point back to the shared cache map (see the sidebar, Examining the Cache Map).  This ultimately caused the Cache Manager to dereference a NULL pointer, and the system crashed.

Examining the Cache Map In order to "peer into" the shared cache map, we relied upon the kernel debugger, which makes it simple for us to examine these key data structures.  The reason for this is that the data types necessary are stored in the PDB files exported by Microsoft (and available on the public symbol server).   Also, keep in mind that this information is not available for older O/S versions.   To look at the shared cache map we use the debugger command "dt nt!_SHARED_CACHE_MAP."  If we want to look at a specific instance, we can provide the address of the shared cache map.  For any file on which caching has been established, this address is stored in the SharedCacheMap field of the SectionObjectPointers structure.   The other key data structure we peek into is the "control area."

 
Our initial theory posited that there was some sort of reference counting problem--after all, the file object should not "go away" or be scrubbed until after the file has been closed.  We turned our attention to the use of "stream file objects" that was made by this particular file system filter driver.  While we couldn't find anything obviously wrong with the use of stream file objects, we did not have much else to "grasp onto."

What we did determine is that if all the reads from this filter were done using non-cached I/O, the problem did not manifest--but the performance of the system suffered dramatically as a result.  Clearly, our theory that this was a cache interaction issue "made sense" given that the problem went away when we disabled caching.  This meant we needed to find the source of the underlying problem, and find a way to provide correct behavior without sacrificing performance.

Our next step was to add logic to perform non-cached I/O when caching had not been previously established for the file; this ensured that the stream file object was not used to back the cache.  Initial testing of this solution suggested it resolved the problem.  Unfortunately, subsequent testing indicated there were still some cases in which this undesirable condition would still occur.

Further analysis finally identified that the earlier presumption--that this involved the stream file objects--was likely faulty.  All testing had been done with various anti-virus products installed (a mandatory configuration for this particular project), and after additional observation we ascertained that the cause was due to performing read operations in the IRP_MJ_CLEANUP handler.  The process layering was that the anti-virus filter was first, followed by the driver under study, then finally the file system driver.

After considerable study, we determined that the anti-virus product was calling IRP_MJ_CLEANUP as part of a file open cancellation.  This filter would then perform a read against the file object and establish caching.  Upon return to the anti-virus filter, that filter would then issue the IRP_MJ_CLOSE and "scrub" the file object.  Afterwards, when the Cache Manager background thread accessed that file object, it would crash if caching had not been re-established on this newly scrubbed file object.

Having identified the specific issue, the resolution was reasonably simple (we forced cache teardown on the file object) and eliminated that class of crash issues.  However, this led us to revisit the numerous issues around canceling an IRP_MJ_CREATE, as well as our previous observations about performing I/O using a passed-in file object.  If anything, this experience reinforced our fundamental belief that a file system filter that wishes to perform I/O--particularly in the IRP_MJ_CREATE, IRP_MJ_CLEANUP or IRP_MJ_CLOSE paths--must do so using their own file object.  

Figure 1, below, shows a code fragment that we wrote while constructing a test scenario.  [Please note that it is provided as a sample.  Try to understand the underlying mechanism before using the "cut and paste" programming technique.  In short, while we believe this code sample should work, as written, it has not been exhaustively tested in the "real world."]  We wanted to validate that the basic idea of using a stream file object always works for cached I/O (see sidebar Unrelated DDK Doc Bug re: a bug found in the DDK docs while writing our test scenario). 

This situation has reinforced our conviction that filter-to-filter interactions remain some of the most complex to properly code, handle, and resolve.  Avoiding such interacts requires a thorough understanding of the underlying Windows system, particularly the interaction between file system drivers, cache manager, and memory manager (see sidebar, Speaking of Understanding Interactions...).  Frequently, drivers are tested in the absence of all other drivers and such interoperability errors are only detected during beta testing or after release testing.

To protect against this class of problems, we suggest that you take the following steps: 

Unrelated DDK Doc Bug In the process of writing our code sample, we did find a bug in the DDK documentation regarding ObReferenceObject and ObDereferenceObject.  These functions are documented as being VOID, yet they actually return the value of the reference count prior to the operation being performed.   We have logged this documentation bug with Microsoft and expect it to be fixed in a subsequent release of the DDK.