Having worked with file system filter drivers in various versions of Windows NT, we have noticed some new tools and issues for Windows 2000 file system filter driver writers that have an effect on their development.  This article attempts to capture those observations in hopes that they will help those developing, or attempting to develop, file system filter drivers in the future.

The “big issues” as far as we can tell continue to remain pretty much the same for file system filter driver writers:

·    Reentrant operations

·    IRQL levels

·    Mapping between file objects and files

·    Dealing with removable media

·    Dealing with naming

·    Understanding file system semantics

Some things have changed in Windows 2000, however, and we thought it was time to discuss some of those issues (and review a few old issues in the process) in The NT Insider so that we can share some of our hard-won experience with others

Device Attachment

Figuring out how to manage device attachment has always been a problem in Windows NT.  For example, there is the perennial problem with respect to load ordering of two file system filter drivers.  In Windows 2000 this problem is less difficult than in previous Windows NT versions, because the CIFS/SMB network redirector now registers as a file system.  Thus, its loading can be detected by a file system filter driver.

In earlier versions of Windows NT, it was necessary to “find out” when a network redirector loaded using some other scheme.  Even in Windows 2000, this problem remains for those filter drivers that filter the NCP (NetWare) network redirector, because it does not register as a “file system”.  Of course, these network redirectors work fine, whether they register as file systems or not, because the only other use the OS has for registered file system is to mount media volumes – and network file systems do not mount media volumes in any case!

Fortunately, the solutions available in previous releases of Windows NT (a timer based solution or filtering and monitoring registration with MUP) continue to work correctly in Windows 2000 and thus they continue to handle the Netware redirector.  For those vendors who have filter drivers that support multiple mechanisms for detecting these file systems, beware: your previous scheme may now function incorrectly in Windows 2000 because you might attach to the CIFS/SMB redirector twice – once when you discover it the “old way” and once when you discover it via a file system registration.

Our solution to this has been to check, each time we try to attach to a volume, to ensure that we are not already attached.  Thus, we can aggressively try to find volumes, knowing that we will only attach once to the corresponding volume. Figure 1 shows the function we use to check for this condition.

//

// CheckForFiltering

//

//  This function looks at an existing device object to ensure that we are not filtering it

//  already.

//

// Inputs:

//  DeviceObject - this is the device we are checking; it should be the base FS object.

//

// Outputs:

//  None.

//

// Returns:

//  TRUE - the device is already being filtered

//  FALSE - the device is not being filtered

//

// Notes:

//  This code must be called before we do an IoAttach* call to ensure we are not

//  creating a duplicate device object.

//

static BOOLEAN CheckForFiltering(PDEVICE_OBJECT DeviceObject)

{

    PDEVICE_OBJECT topDeviceObject = DeviceObject;

    //

    // We are going to walk the attached device chain to see if we can find an attached

    // device that is ours

    //

    while (topDeviceObject) {

        POUR_DEV_EXT ourExt;

        ourExt = (POUR_DEV_EXT) topDeviceObject->DeviceExtension;

        if ((NULL != ourExt) &&

            (OUR_EXTENSION_MAGIC_NUMBER == ourExt->ExtensionMagicNumber)) {

            //

            // This is OUR device object, so we are already filtering this device

            //

            return TRUE;

        }

        //

        // This was not our device; go to the next one in the chain.

        //

        topDeviceObject = topDeviceObject->AttachedDevice;

    }

    //

    // If we get to this point we know we are not filtering this device.

    //

    return FALSE;

}

Figure 1

Thus, prior to attaching to the device, we always check to see if we are already attached to the device.

Another issue with respect to device attachment is not specifically a Windows 2000 issue, since this change occurred in Windows NT 4.0, but we think it is still worth mentioning.  Rather than using IoAttachDeviceByPointer, a file system filter driver should use IoAttachDeviceToDeviceStack.  This is because there is a narrow race condition that can arise when two file system filter drivers are attempting to attach to the same underlying device – they will be layered with respect to one another, but may not be aware that the one on top must call the lower filter.  This can lead to incorrect filter behavior.

While it is possible to solve this “race condition” while still using IoAttachDeviceToDeviceStack (and we had to do this for early versions of Windows NT,) we encourage use of the new function, rather than the older, deprecated function.

Filter Interactions

While not new in Windows 2000, filter-to-filter interactions are a significant problem.  Windows 2000 exacerbates these problems both because more third-party developers are using file system filter drivers and because Microsoft has introduced new features that encourage the use of more file system filter drivers.  In addition, they have added filter drivers that take advantage of those new features, and thus, for the first time, Microsoft itself ships file system filter drivers!

Why are filter interactions an issue?  We’ve seen several general “categories” of issues for file system filter drivers interacting with one another:

·    Functional Interference – This is the case when the base function of one filter interferes with the base function of another.  For example, an encryption filter and an on-access virus scanner can interact with one another to cause substantial problems.

·    Reentrant Interference – This is the case when one filter uses one mechanism for detecting reentrant calls, while another filter uses a different mechanism for detecting reentrant calls.  Sometimes these two mechanisms are compatible, but it is quite easy to construct mechanisms that are not safe when two filters are using reentrancy.

·    Incorrect or Partial Implementation – For file system filter drivers, the Fast I/O routines can quickly become a point of contention if one file system filter driver implements a fast I/O routine and another does not.  If the filter at the top of the device stack does not implement fast I/O, the I/O Manager will “skip” or “bypass” the filter driver and call the underlying file system directly.  Alas, this bypasses the properly implemented filter as well.

·    Locking behavior – We routinely see cases where the use of locks within a file system filter driver can cause problems when a second file system filter driver becomes involved in the process.  For example, if one file system filter driver uses a “fast mutex” and then calls a Zw routine (e.g., ZwReadFile) it is quite possible that a second filter that chooses to post the I/O operation (by marking the IRP as pending and returning STATUS_PENDING) will cause the I/O to never complete.  If either filter is loaded, the system works correctly. Only when BOTH filters are loaded does the system hang.

·    Timing behavior – The addition of a new filter, with new locks, new IRP management queues, etc. will impact the timing and change of the flow of I/O in the system.  Any time you change the timing behavior, you risk uncovering new bugs – especially within the file system filter drivers (which typically have far less exposure to hostile environments).

·    Stack Overflow – The addition of multiple file system filter drivers sometimes leads to cases where there is an inadequate amount of stack space.  Recall that kernel stack is typically limited to 12KB.   Given the reentrant nature of file systems, as well as the reentrant nature of filter drivers, it is easy to find cases where there just isn’t enough stack space.

Unfortunately, there is no single solution to these problems.  Further, sometimes the solution to one problem can exacerbate a different problem.  For example, one technique for dealing with stack overflow is to send a “work item” to a worker thread, so that the worker thread can complete the processing (using a different stack).  That works fine until you factor in locking.  Again, techniques that work with one file system filter driver frequently do not work with two (or more).

IRP Management

Windows 2000 has introduced a number of new calls for handling I/O request packets that are geared towards file system filter drivers.   This includes IoCopyCurrentIrpStack LocationToNext and IoSkipCurrentIrpStackLocation.

IoCopyCurrentIrpStackLocationToNext copies the parameters of the current stack location to the next stack location.  It does so without copying other information, notably the caller’s I/O completion routine, which can cause incorrect function.

IoSkipCurrentIrpStackLocation simply modifies the I/O stack so that the current stack location is re-used when calling the next driver.  This is useful if no completion routine is needed for the current driver because it eliminates the data copy and the completion processing associated with this driver.

Unfortunately, the quintessential example of constructing a file system filter driver in Windows 2000, sfilter, does not utilize these routines.  For example, the Windows 2000 IFS Kit includes the sample code in the routine SfCreate (See Figure 2).  Were this routine to be written with the newer Windows 2000 functions it would look like Figure 2a.

//

    // If debugging is enabled, do the processing required to see the packet

    // upon its completion.  Otherwise, let the request go w/no further

    // processing.

    //

    if (SfDebug) {

        PIO_STACK_LOCATION nextIrpSp;

        //

        // Simply copy this driver stack location contents to the next driver's

        // stack.

        //

        nextIrpSp = IoGetNextIrpStackLocation( Irp );

        RtlMoveMemory( nextIrpSp, irpSp, sizeof( IO_STACK_LOCATION ) );

        IoSetCompletionRoutine(

            Irp,

            SfCreateCompletion,

            NULL,

            TRUE,

            FALSE,

            FALSE

            );

        }

    else {

        Irp->CurrentLocation++;

        Irp->Tail.Overlay.CurrentStackLocation++;

        }

Figure 2

//

    // If debugging is enabled, do the processing required to see the packet

    // upon its completion.  Otherwise, let the request go w/no further

    // processing.

    //

    if (SfDebug) {

        PIO_STACK_LOCATION nextIrpSp;

        //

        // Simply copy this driver stack location contents to the next driver's

        // stack.

        //

        IoCopyCurrentIrpStackLocationToNext(Irp);

        IoSetCompletionRoutine(

            Irp,

            SfCreateCompletion,

            NULL,

            TRUE,

            FALSE,

            FALSE

            );

        }

    else {

        IoSkipCurrentIrpStackLocation(Irp);

        }

Figure 2a

While either code sample works correctly, the second example takes advantage of the Windows 2000 support for these new functions.

Of course, we note that in fairness, these two functions are macros.  Thus, because we must support both Windows 2000 and Windows NT 4.0, we have back-ported these macros into our Windows NT 4.0 code by simply adding the code in Figure 3.

#if !defined(IoCopyCurrentIrpStackLocationToNext)

#define IoCopyCurrentIrpStackLocationToNext( Irp ) { \

    PIO_STACK_LOCATION irpSp; \

    PIO_STACK_LOCATION nextIrpSp; \

    irpSp = IoGetCurrentIrpStackLocation( (Irp) ); \

    nextIrpSp = IoGetNextIrpStackLocation( (Irp) ); \

    RtlCopyMemory( nextIrpSp, irpSp, FIELD_OFFSET(IO_STACK_LOCATION, CompletionRoutine)); \

    nextIrpSp->Control = 0; }

#endif

#if !defined(IoSkipCurrentIrpStackLocation)

#define IoSkipCurrentIrpStackLocation( Irp ) \

    (Irp)->CurrentLocation++; \

    (Irp)->Tail.Overlay.CurrentStackLocation++;

#endif

Figure 3

Which is, of course, nothing more than the Windows 2000 definitions of these functions.

The I/O Manager has always maintained two “pools” of pre-created IRPs and Windows 2000 is no exception here.  One of these pools has a single I/O stack location.  The other pool has seven IRP stack locations in Windows 2000 (versus four in Windows NT 4.0).  This is reflective of the fact that the driver stack is now larger, mostly due to the introduction of the bus driver/function driver abstraction used in supporting Plug and Play.

Of course, the OS will function properly if a particular driver stack needs more than seven IRP stack locations, but it must allocate those IRPs from non-paged pool rather than from the I/O Manager free list, and thus this increases the overhead for each I/O operation.  Of course, this is another situation where multiple file system filter drivers can complicate matters, because each new driver increases the necessary number of I/O stack locations!

One other new function that can be useful for file system filter driver writers is IoCancelFileOpen.  This function was added to allow a file system filter driver to reverse the effects of a previous IRP_MJ_CREATE that was processed successfully by the underlying file system.  Essentially, this sends both an IRP_MJ_CLEANUP and IRP_MJ_CLOSE IRP to the underlying file system as well as resetting state for the newly opened (and now closed) file object.

Prior to the introduction of this call in Windows 2000, a file system filter driver needed to perform this ancillary clean-up processing directly (by building and sending these IRPs, as well as resetting the file object state).

Removable Media

Very few things can cause more problems than removable media, and this problem is no better in Windows 2000 than it is in earlier versions.

For example, one problem that we continue to observe is a bug where the filter driver deletes its device object prematurely.  For this to occur, the filter driver must use a completion handler.  The call arrives in the filter driver.  The filter in turn passes the call down to the underlying file system. The underlying file system device then deletes its device object, which in turn causes the filter’s fast I/O detach device entry point to be called.  The filter driver then detaches and deletes its own device object.

The file system then completes the IRP (normally an IRP_MJ_CLOSE).  The I/O Manager then calls the filter’s completion routine.  The completion routine, if it attempts to access the device object, can experience various problems – since it has already deleted that device object!

For example, if we look at the FileSpy code in the IFS Kit, we note that in their completion routine they access their device object (in SpyPassThroughCompletion):

if (SHOULD_LOG(DeviceObject))

And of course the corresponding (and necessary, to trigger this problem,) code in DetachDevice:

    IoDetachDevice( TargetDevice );

IoDeleteDevice( SourceDevice );

(Indeed, you can find comparable code in the sfilter example as well).

Now, if we review FastFat, we note that the routine FatCheckForDismount is called when there are no longer any open files on the volume (that is, this is the last close on the volume).  This is in close.c, near the end of FatCommonClose (See Figure 4) and then in FatCheckForDismount (See Figure 4a).

if ( (Vcb->OpenFileCount == 0) &&

                     ((Vcb->VcbCondition == VcbNotMounted) ||

                      (Vcb->VcbCondition == VcbBad)) &&

                     FatCheckForDismount( &IrpContext, Vcb, FALSE ) ) {

                    //

                    //  If this is not the Vpb "attached" to the device, free it.

Figure 4

        FatDeleteVcb( IrpContext, Vcb );

        Vpb->DeviceObject = NULL;

        IoDeleteDevice( (PDEVICE_OBJECT)

                        CONTAINING_RECORD( Vcb,

                                           VOLUME_DEVICE_OBJECT,

                                           Vcb ) );

        VcbDeleted = TRUE;

Figure 4a

Then (back in FatCommonClose) the processing finishes and (in FatFsdClose) the file system calls FatCompleteRequest.  This then calls into the filter driver’s completion routine, where it references its now deleted device object’s device extension.

As we say around here at OSR – “I HATE when that happens!”

We’ve solved this by implementing a reference counting scheme, so that each time we send an IRP down to the underlying file system, we bump a reference count in the device object extension.  When we receive the device back in the completion routine we decrement the reference count.  We also maintain one reference for the attachment.  Thus, when the device is both detached and there are no outstanding IRPs, then we delete the device object.

While this is additional “overhead” it means that periodic blue screens when using removable media can be eliminated.

In addition to “removable media” (in the sense of floppies, etc.) Windows 2000 also adds the ability to have devices that can be removed (rather than the media within the device).  We discuss the issues related to removing the device a bit later when we describe plug and play issues.

Managing File Objects

Few things continue to confound file system filter driver writers more than the issue of how to handle file objects within their filter driver.  In Windows 2000, the introduction of a new function, IoCreateStreamFileObjectLite has made their task even more difficult than it was in the past.

Just to remind all of the file system filter driver writers: File Objects are not files. They are references to the actual file.  Each time the file is opened, a new file object is created.  Applications can open files.  Filter drivers can open files.  File Systems can open files.  Many files are opened via IoCreateFile (this is the path that is used by applications and many kernel mode components).  A few files are opened using IoCreateStreamFileObject and IoCreateStreamFileObject Lite.

Of course, a typical file system filter driver associates file objects together by using the FsContext field.  While this works with all file systems, we note that in the case of network file systems the values of this field are often updated throughout the life of the file.  This is because a network file system may not actually “open” the file until an application begins to perform I/O against the file.  We have even observed some redirectors that leave this field as NULL after the IRP_MJ_CREATE, only filling it in later, when they have the necessary information.

Probably the greatest difficulty facing file system filter driver writers are the file objects created by the file system itself.  That is because normally the creation of a new file object is indicated to the filter driver via an IRP_MJ_CREATE but for file objects created by the file systems directly, there is no such indication.

In Windows NT 4.0, filter driver developers found that they did receive an IRP_MJ_CLEANUP for these file objects (created using IoCreateStreamFileObject) but in Windows 2000, the new call IoCreateStreamFileObjectLite eliminates even the IRP_MJ_CLEANUP call.

To work around this, we normally suggest that file system filter driver writers organize their data structures so that they maintain a per-file structure keyed on the FsContext value.  Within each of the per-file structures, we then maintain a list of each file object associated with that file.

Then, during the processing of an IRP_MJ_READ, IRP_MJ_WRITE, IRP_MJ_QUERY_INFORMATION, IRP_MJ_SET_INFORMATION, IRP_MJ_CREATE or IRP_MJ_CLEANUP operation we check to see if this file object is currently listed as one of the file objects associated with the given file.  If it is not associated with the given file structure, we then associate it.  This ensures that we will not discard the per-file structure prematurely.

Some of these files are internal control files for the FSD itself, and in that case your filter may not be tracking a per-file data structure for them at all.  In most cases we ignore those files, but if they are important to your specific filter driver, this mechanism will allow you to detect and track them as well.

For network redirectors, we suggest monitoring the FsContext field before, and after, an I/O operation is processed by the underlying FSD; then, if the value of this field changes across the call, the filter can update its data structures accordingly.

Plug and Play

The introduction of plug and play in Windows 2000 has only minimal impact on the development of file system filter drivers, as it turns out.  Still, for some file system filter drivers it is useful to track the various plug and play states.

Fortunately, for file systems, there are only a few operations that are normally of interest:

IRP_MJ_CANCEL_REMOVE_DEVICE – this indicates that the previous request to remove the device has been cancelled.  The device will not be removed.

IRP_MJ_REMOVE_DEVICE – this indicates that the device has been removed.  Normally, once the underlying device driver has confirmed the removal of the device, subsequent I/O operations would fail.

IRP_MJ_SURPRISE_REMOVAL - this indicates that the device was removed without any previous indication that this would occur. 

Virtual Memory

Very few things confound file system filter driver writers than the way that the file systems interact with the virtual memory system.  In Windows 2000, one of the new surprises has been the increasing use of memory-mapped files.

The “notepad” utility was modified so that it now utilizes memory mapping to modify files.  In Windows NT 4.0 there were file system filter drivers that ignored the issues of memory mapped files because “nobody uses them.”  This is no longer the case in Windows 2000!

Just as in Windows NT 4.0, files may be modified directly in the cache by utilizing the “memory mapped file interface.”  Modifications made through the virtual memory interface are not seen via normal IRP_MJ_WRITE operations from the application because they are made, via direct memory references, to the copy of the data in the cache itself.  Thus, the file system is not involved in this operation at all.  The changes, however, must eventually be sent to the underlying file system so that they may be committed to persistent storage.

For a file system filter driver, these changes are detected as paging I/O operations – the IRP_PAGING_IO bit is set inside of the IRP_MJ_WRITE IRP that arrives in the file system.  Of course, the “issue” for a filter driver is that applications using the normal read/write interface will indicate their operations using IRP_MJ_WRITE and then those modifications will be made (by the file system) against the copy in the cache.  That data is then written to the cache by the file system.  Later, the contents of the cache (just like with memory mapped files) are written back to the file system for actual storage!

Of course, one of the “complications” of tracking paging I/O is that this, combined with the use of dummy file objects by the file systems, leads to the interesting case where the paging I/O is done against one of these internally created file objects.

The drawing shown in Figure 5 is one that we use when describing this phenomenon in our file systems class.

Figure 5

As you can see, there are multiple file objects referencing the same common structures (the “file control block” as it is typically called and the VM control structure, the section object pointers). 

The interesting point, however, is that the VM section object refers back to one specific file object.  Whenever the VM system needs to perform an operation against that section, the paging I/O operations will be issued against that specific file object.

In the notepad example, it is quite possible for notepad to open the file and yet have all I/O operations occur against a file object other than the one created for notepad.  This can definitely confuse an unsuspecting file system filter driver writer!

This specific behavior is actually quite common for NTFS, because NTFS routinely creates its own internal file objects.  If those file objects are, in turn, used for backing the section object, all subsequent paging I/O operations will be done against that file object.  If a file system filter driver does not track this file object (because it never observed the IRP_MJ_CREATE) then it will not properly handle this I/O!

Beyond Windows 2000

Future versions of Windows NT (beyond the current Windows 2000 release) will incorporate yet more changes for file system filter driver writers.  For example, Microsoft has indicated at recent IFS Interoperability plug-fests that they are working towards a new file system filter driver model that will greatly simplify the task of writing a file system filter driver – and encourage better interoperability between file system filter drivers.

For now, though, file system filter driver writers have a new set of tools and issues to address as they build their Windows 2000 file system filter drivers!