Figure 1 - Backtrace on the kbdus.dll Nightmare

The name "ConvertHandleAndVerifyLoc" was suggestive that this was in some fashion name sensitive. Recall that we are redirecting to a parallel stack, so the device name will be different, even if the path and file name are the same. As we're so fond of saying around here, "Names don't really mean much in Windows", but apparently that wisdom hasn't sunk in everywhere.

With a bit more work in the debugger, watching the behavior of this particular function, we were able to confirm that in fact it takes the file object and queries the name and then compares that to the name that it computes. Thus, we ended up with our name: \Device\<our parallel device>\Windows\System32\kbdus.dll and Win32 computed that it should be \Device\HarddiskVolume1\Windows\System32\kbdus.dll. When the two didn't match, it did the terminal exit.

This presented an interesting challenge, since we do not control the name of our device. Thus, there was no immediately obvious way to "make this work." The solution that we ultimately employed relies upon the fact that each file object contains a VPB and the VPB within the file object need not be the same as the VPB in the media device volume. Previously, we had not been using a VPB (since we're a parallel stack, we look more like a network file system). However, we decided that for physical volumes we would create a VPB. That VPB points to our parallel file system device object and to the underlying media volume.

That now means that when Win32 queries the name, the I/O Manager will know to extract the name of the underlying media volume and do a name query to the file system. Since it follows FileObject->Vpb and not DeviceObject->Vpb , it gets the right device name back.

Very few applications are sensitive to the actual name of the underlying device on which they are located. If we ignored this DLL, the system booted and ran without incident. Thus, that it was so sensitive for this specific file surprised us. Our only theory was that this is an attempt in Windows Vista to attempt to detect and prevent key loggers.

One of the big changes of which we were aware in Vista was the decision to move the Filter Manager's attachment from the individual redirector to the MUP aggregation device. We have previously observed that this does create strange issues for legacy filters (e.g., our own legacy filter kit, which predates the callback model and watches for filters both by name and by their registration with MUP). We suspected that we would find other subtle issues as well and we have not been disappointed.

A common technique in file systems and filter drivers is to create stream file objects via IoCreateStreamFileObjectEx (or the other two variants of this function). We do this because in our parallel file system stack we need a file object to use when interacting with the underlying file system - in this case MUP.

1: kd> !analyze -v
*******************************************************************************
*    *
* Bugcheck Analysis  *
*    *
*******************************************************************************

MUP_FILE_SYSTEM (103)
MUP file system detected an error.
Arguments:
Arg1: 00000001, MUP_BUGCHECK_NO_FILECONTEXT
Could not locate MUP file context corresponding to a file object.
Arg2: 9a364f00, Irp Address
Arg3: 83545e38, FILE_OBJECT Address whose MUP file context could not be found
Arg4: 83cb7c68, DEVICE_OBJECT Address

Debugging Details:
------------------

This was an error path case - for whatever reason, the open on the file had failed and we had thus decided to tear down the file object. Since the file object points to MUP, the IRP_MJ_CLEANUP operation is passed to MUP and it then dies a horrible death because there is no FsContext value. Unfortunately, this behavior differs dramatically from what the other file systems - and notably those in the WDK - actually do in this case. Here's the relevant code from the FAT source code in the WDK (See Figure 3, below).

   //
    //  Special case the unopened file object.  This will occur only when
    //  we are initializing Vcb and IoCreateStreamFileObject is being
    //  called.
    //

    if (TypeOfOpen == UnopenedFileObject) {

        DebugTrace(0, Dbg, "Unopened File Object\n", 0);

        FatCompleteRequest( IrpContext, Irp, STATUS_SUCCESS );

        DebugTrace(-1, Dbg, "FatCommonCleanup -> STATUS_SUCCESS\n", 0);
        return STATUS_SUCCESS;
    }

The description is on target here - this is an unopened file object that is being torn down because something else has gone wrong. We maintain that this really is a defect in MUP, since this behavior does not manifest in any of the actual file systems (and they all handle this case in similar fashion).

Working around this issue proved to be a bit painful. We needed to dereference the file object to make it go away in this case but we could not have it point to MUP. Thus, we decided to have it point to our device object. This just left the "small detail" of reference counts which, unfortunately, required that we do something rather ugly (after all, we're manipulating I/O Manager reference counts). Here's one of the code segments that we used for handling this issue (See Figure 4, below).

                if (pFileObject->DeviceObject &&
                    (DmkGetMupDevice()== pFileObject->DeviceObject)) {
                    LONG refCnt;
                    KIRQL oldIrql;

                    //
                    // MUP does not properly handle the IRP_MJ_CLEANUP.  This logic
                    // works around this issue but relies on several things:
                    //  - That MUP isn't really going to "go away" (unload), so the
                    //    ref count won't go to zero.
                    //  - That setting the device object to NULL will just cause the
                    //    FO to be deleted (no irp)
                    //
                    // Note that this is really an issue on Vista where the filter
                    // sits on top of MUP, versus earlier versions where it sat on
                    // the redirector.
                    //
                    // This is a VERY ugly fix - while we serialize "properly" we
                    // cannot do teardown here.  If the ref count ever goes to zero
                    // we will need to rethink this whole process.  Hopefully, we can
                    // convince Microsoft that this is a bug worthy of being fixed.
                    //
                    //
                    oldIrql = KeAcquireQueuedSpinLock(LockQueueIoDatabaseLock);

                    //
                    // Bump our own ref count.
                    //
                    DeviceObject->ReferenceCount++;

                    //
                    // Drop the ref count on the original (MUP) device
                    //
                    pFileObject->DeviceObject->ReferenceCount--;
                    refCnt = pFileObject->DeviceObject->ReferenceCount;
                    KeReleaseQueuedSpinLock(LockQueueIoDatabaseLock, oldIrql);

                    //
                    // Now we can point this file object at our own device, since
                    // we handle it properly in teardown.
                    //
                    pFileObject->DeviceObject = DeviceObject;

                    //
                    // Since we don't believe we'll ever make this go to zero, let's
                    // assert it here in case we're wrong.
                    //
                    ASSERT(refCnt);
              
                }

In our own parallel file system stack, we properly handle the IRP_MJ_CLEANUP in this case. This effectively resolved this particular problem.

Conclusion

Our experiences to date have reinforced for us the understanding that Windows Vista is a major new version of Windows. It includes numerous changes, often subtle, that impact directly on file systems and file system filter drivers. Some of these can be easy to work around, others are far more complex.

The lesson for everyone in this field is to keep in mind that your file systems and file system filters will observe different behavior in Vista than they did in earlier OS versions. The challenge is thus to find them and resolve them in a fashion that is applicable to all the platforms that you support.