One of the challenges for a file systems filter driver in the Windows NT, 2000 or XP environment is knowing precisely when it is “OK” to discard per-file state information.  While it is customary for file system filter drivers to track such state based on the FsContext field of the FILE_OBJECT, the new file system filter driver writer may be frustrated to find that some I/O operations are missed.

Analyzing this problem, it is easy to observe that matching IRP_MJ_CLOSE operations against IRP_MJ_CREATE operations is not enough to properly determine when it is OK to discard the FsContext-related file state within the filter driver. We have previously described why this is the case (the use of stream file objects by the file systems) but in this article the emphasis is on describing an algorithm that allows the filter driver not to discard the state until the last reference to the file is truly gone.

First, it is worth reviewing the reason that reference counting is not a trivial exercise for file system filter drivers, then we’ll review the various cases that any algorithm must take into account, and finally propose our algorithm for tracking references to the per-file data normally maintained by a file system filter driver.

A file system has several mechanisms for creating file objects that it may use for its own internal processing.  These include IoCreateStreamFileObject, IoCreateStreamFileObjectLite, and IoCreateStreamFileObjectEx.  Not all of these are available in all versions, but even the Windows NT 4.0 IFS Kit contained the IoCreateStreamFileObject call.  Both IoCreateStreamFileObject and IoCreateStreamFileObjectLite are documented in the Windows XP IFS Kit and each provides similar functionality – they create file objects based upon either an existing file object or device object.  The IFS Kit documentation sums up the problem for file system filter drivers, in the page for IoCreateStreamFileObjectLite:

“Thus filter drivers should expect to receive IRP_MJ_CLOSE requests for previously unseen file objects.”

Of course, the fact that a file system filter driver receives an IRP_MJ_CLOSE operation for file objects which were never indicated to the filter driver does complicate reference counting.

In looking at the sequence of events, we note that we are concerned about the overlap of the IRP_MJ_CREATE and IRP_MJ_CLOSE of normal user access along with the IRP_MJ_CLOSE of file system access.  Thus, we might observe the following scenarios:

• An application program opens the file and then closes it; the filter never observes any stream file objects used against the given file.
• The filter only observes stream file objects and does not observe any application access to a file.
• The application opens the file and the filter observes both application and stream file objects used to access the file in question.
• The filter observes stream file object usage for a file, and then subsequently observes an application open the file.

Whatever algorithm used by the file system filter driver to track this context and state information, it must be able to account for all of these scenarios.

In analyzing the various algorithms available for tracking this information, we noted that there may be cases where the techniques used can be simplified.  And thus we describe two separate algorithms here – one that is useful for tracking state in the general case, and a second case that is useful when only data operations are of interest to the file system filter driver.

For a file system filter driver that must track all instances of a file object, this unfortunately results in the need to not only track reference counts, but also to track the specific file object for which it has observed and IRP_MJ_CREATE.  Such a filter must typically maintain both a reference count (for file objects seen via the IRP_MJ_CREATE mechanism) and an explicit list (or other data structure for tracking these file objects) of stream file objects seen by the filter.  Thus, on each I/O operation, the filter examines the FO_STREAM_FILE bit in the Flags field of the FILE_OBJECT.  If this bit is set, the filter verifies that the file object is in its list – if not, it allocates the necessary storage for its tracking information and inserts it into its list.  Then, when it observes an IRP_MJ_CLOSE operation, it will either decrement the reference count or remove it from the list, as appropriate.  Only when both the list is empty and the reference count is zero does the filter delete the tracking data structure.

Thus, the filter must create the per-file tracking information whenever it observes an IRP_MJ_CREATE or a stream file object for a file for which it does not have tracking information, as a file object may be created entirely internal to the filter driver or because of an application program’s request to open the file.  The filter driver must still attempt to determine the appropriate action with respect to tracking such files, which can be difficult.  In the “simplified” algorithm that we describe later, we will suggest a technique for keeping the context even longer to allow the filter to avoid re-evaluating the need to track the given file.

There are four distinct scenarios we described earlier for consideration with this algorithm:

• If the application program first opens the file and then closes it, with no intervening stream file object usage, this algorithm works because we will delete the file context when the reference count drops to zero.  There is no stream context, so the list will be empty.
• If only stream file objects are observed, the filter driver might potentially only see an IRP_MJ_CLOSE operation for this file object, in which case the file was not of much interest.  Otherwise, on observing the first I/O operation using the stream file object the filter would create a context record and insert tracking information for the given file object.  When it observes the subsequent IRP_MJ_CLOSE operation, it will remove the entry from the list and delete the file context block.
• If an application opens the file initially, the filter driver would create the per-stream file context information at that point. If it then subsequently observes an I/O using a stream file object, it will not find it in its list, and will create a new entry.  Then, regardless of the order in which the IRP_MJ_CLOSE operations arrive within the filter driver, it will not delete the per-file context until they have both been processed – the reference count will then be zero and the list of stream file objects will be empty.
• If the stream file object is first observed, the filter driver will create its per-file context information at that point and insert a tracking block for the stream file object into the list. A subsequent application open of the file will cause the filter to increment the reference count.  Again, until both the stream file object list is empty and the reference count is zero, the per-file context information will be maintained.

Of course, there are numerous variations on these basic scenarios, involving multiple stream file objects and multiple application opens.  However, we have focused on the boundary cases because we maintain that additional stream file objects or application opens will work in a comparable fashion, albeit with additional entries or references.

Figure 1 — Section Object/File Object Relationship

Many filter drivers can use a simpler (and less costly) algorithm because they only care about data operations for user files.  In this case, we note that the only issue is when stream file objects are used to back a section object for the memory manager.

This is because any paging I/O operations that the memory manager performs will be based upon the file object referenced by the given section object.  The risk (for the filter driver) is that it will observe the following sequence:

(1)     The application opens the file;

(2)     The file system creates a stream file object and that object is then used as the file object to back the section (i.e. a call to CcInitializeCacheMap is made using that stream file object).

(3)     The application closes the file.  Because there are no outstanding references (other than the application handle) both IRP_MJ_CLEANUP and IRP_MJ_CLOSE are sent to the file system (and hence through the filter).  The filter deletes its per-file context information.

(4)     The Memory Manager flushes any changes to the data (perhaps through the section, as might be the case for memory mapped file access) back to the file system using paging I/O operations.

The problem with this scenario is that because the filter has deleted its per-file context information, the subsequent write operations are missed by the filter because they do not appear to be relevant.  Of course, using the previous algorithm we described, upon observing the new paging I/O operation the filter would create the per-file context information (again) and then ascertain if the file should have been tracked in the first place.  While this will work, we can simplify the process by adding an additional check for this case – specifically, before deleting the per-file context, the section object pointers from the file object must both be NULL.  Otherwise, we know that there is some file object backing the section and it has not yet been observed by our filter driver.

Thus, for a filter driver that is concerned about all I/O operations and tracking all file state, this can be added to the previous algorithm to keep state information when it still might be useful.

For a filter driver that is only concerned about data operations (read and write) this suggests a simpler algorithm:

• For each IRP_MJ_CREATE, the filter driver increments the reference count on the per-file context structure.
• For each IRP_MJ_CLOSE the filter driver decrements the reference count on the per-file context structure if the FO_STREAM_FILE bit is not set for the file object and if CcGetFileObjectFromSectionPtrs does not return this file object.
• If the reference count for the per-file context structure reaches zero and both the ImageSectionObject and DataSectionObject of the SectionObjectPointers field from the FILE_OBJECT is zero, the filter driver may then delete the per-file context data.

This algorithm avoids the need to track individual file objects.  It deals with stream file objects by ignoring any of them that are not involved in paging I/O.  For many filter drivers, this solution will be sufficient for them to track their per-file state.

This works because the reference count represents the user-level opens against the file.  The reference count is decremented only when a user-level file object is closed (as evidenced by the FO_STREAM_FILE bit’s absence in the file object) and thus there are no user references to the file when the reference count is zero.  Similarly, the section objects represent the VM state for the file.  If these two section object pointers are NULL there are no VM references against the file – hence, no memory mappings of the file, whether by applications or the cache manager.  Since the filter is only interested in data operations, this is sufficient.  Any new data operations will require that the file be re-opened.  Do note that this algorithm only works if your filter observes all operations against this file - if you dynamically load your filter driver and the file has already been opened prior to that, this algorithm will not suffice.

For file system filter drivers that will run only on Windows XP or more recent versions, there is no need for tracking the per-file context information in this fashion.  Instead, the routines FsRtlInsertPerStreamContext and FsRtlRemovePerStreamContext can be used by a file system filter driver to allow the file system runtime library package to manage the per-stream file contexts.  For filter drivers that support both Windows XP as well as earlier versions, these two techniques can be used to implement one common mechanism that works for all platforms.