The Windows XP operating system contains a number of new calls as well as changes to existing calls that may materially impact the development of file systems and file system filter drivers, especially for those porting from earlier versions of the operating system.  The purpose of this article is to describe those changes we believe may have the greatest impact on the file systems and file system filter driver development community.

In describing these, we will focus on several key areas:

·    The new IFS Kit package

·    New APIs for file system filter drivers

·    Changes to existing APIs

·    New Security APIs

·    Other Interesting Changes

There are a tremendous number of small changes, some of them relating to changes in the DDK as well (for example, the support for marking deprecated functions), and we defer discussion of these.  Our focus will be on those changes most material for the file systems community.

New IFS Kit

As might be expected with a new release of the Windows operating system, there is a new IFS Kit.  Anyone who wishes to obtain a copy of this kit must order it from the Microsoft web site (which, at the time we wrote this article, can be found at http://www.microsoft.com/ddk/ifskit).  Unlike its predecessor (the Windows 2000 IFS Kit), it does not contain copies of the earlier kit materials.  Thus, if you must still develop file systems or file system filter drivers for Windows 2000 or even Windows NT 4.0, you also need to have a copy of the earlier kit.

The new IFS Kit contains numerous new materials, not previously provided with earlier kits.  This includes:

·    Two DVDs made from the original IFS Conference presentations from October 1994.  While there have been changes to the OS since then, these presentations provide valuable insight into the internal workings of the operating system itself.

·    Updated sample file system drivers including file systems for FAT, CDFS, and two sample mini-redirectors.

·    Update sample file system filter drivers including both sfilter and filespy.

·    Updated documentation.  Even for those developers not planning on immediately developing Windows XP file systems or file system filter drivers, the documentation in the new kit is invaluable, especially since much of the documentation is new, but the functionality is not.

·    Stand-alone use.  The new kit contains everything you need to build file system filter drivers, including the compiler, linker, DDK environment, and IFS environment.

Those of us here at OSR who work with file systems on a daily basis are quite impressed with the new kit – it clearly shows that the folks at Microsoft have been listening to the feedback from the development community and are working hard to address our concerns.  Kudos to the entire IFS Kit development team for a job well done!

New File System Filter APIs

One of the most significant enhancements is the support provided by the core OS for file system filter drivers.  This new IFS Kit addresses a number of “standard” problems that must be routinely handled by file system filter drivers:

·    Tracking per-file (or “per stream”) context information.  Doing so requires explicit file systems support by using the FSRTL_ ADVANCED_FCB_HEADER (described a bit later in this article), but the standard file systems in Windows XP have been modified so that they provide this support.  We describe this in more detail later in this section.

·    The ability to detect when a file object represents a paging file using FsRtlIsPagingFile, a function that takes the file object and indicates whether or not the resulting file represents a paging file.

·    The ability to filter the six locking callbacks in the fast I/O entry point table.  This is done using FsRtlRegisterFileSystemFilterCallbacks.  We describe this in more detail later in this section.

·    New security routines that allow retrieving the textual name of the user, rather than only the security identifier of the user (see SecLookup AccountName).

·    A new I/O Manager function that may be used to initiate a new file open operation at a specific location in the driver stack, thus eliminating many of the re-entrancy problems faced by file system filter drivers (see IoCreateFileSpecifyDevice ObjectHint).

·    A new I/O Manager function that may be used to synchronously call the underlying driver (Io ForwardIrpSynchronously).

·    A new I/O Manager function that may be used to retrieve a DOS drive-letter style name for a given device (IoQueryFileDosDeviceName).

·    A new I/O Manager function that may be used to “cancel” an open from the filter driver; this works by calling cleanup and close in the lower level file system driver (IoCancelFileOpen).

·    New USN journal functions for managing the USN log.  These new functions allow the filter to disable journaling for a particular file (based upon its handle) and indicate the reason for which this change is not being journaled (USN_SOURCE_ DATA_MANAGEMENT, USN_SOURCE_ AUXILIARY_DATA, and USN_SOURCE_ REPLICATION_MANAGEMENT).

·    New functions to allow filters to watch for directory change notification operations (FsRtlNotifyFilterReportChange and FsRtl NotifyFilterChangeDirectory).

From the perspective of file system filter driver writers, these are all changes that will simplify development issues for file system filters being written to run on Windows XP.  Of all of these changes the two that most materially impact the file system filter driver are those related to per-file context blocks as well as those related to handling file system callbacks for the “special” fast I/O functions.

In many file system filter drivers, the filter driver tracks context on a per file basis.  Typically, in versions prior to Windows XP, this is done using some sort of table-based lookup on one of the file object fields that is maintained by the file system in a per-file basis (e.g., the FsContext field or the SectionObjectPointers field from the FILE_OBJECT structure).  In Windows XP, a file system may use the FSRTL_ADVANCED_FCB_HEADER and indicate that it supports filter contexts (which essentially means that it initializes the advanced header so that the FsRtl package can use it for tracking these context values).  If a file system does support filter contexts (and all of the standard file systems distributed with Windows XP do support them) then a file system filter driver can use several new API calls.  Note that in discussing these APIs the documentation refers to them as “per stream” context values.  For most file systems, there is only a single stream per file and thus this is equivalent to “per file” for such file systems.  Regardless, the data is associated with the FCB header and is thus stored in a one-to-one correspondence with the FsContext field of the file object.

The new APIs are:

·    FsRtlInitPerStreamContext – this routine is used by the filter driver to initialize its per-stream context information with the file.  It requires that the filter allocate the storage for the context information itself, provide space for the FSRTL_PER_STREAM_CONTEXT block in the filter driver’s context structure, as well as a function that the operating system will call when this stream context information is being deleted.

·    FsRtlInsertPerStreamContext – this call is used by the file system filter driver to associate a given context block (allocated by the filter and initialized using FsRtlInitPerStreamContext) with the stream associated with the given file object.

·    FsRtlLookupPerStreamContext – this call is used by the file system filter driver to locate a given context block that is associated with the file object.  Typically, a file system filter driver will identify its own context block using unique OwnerId and InstanceId parameters when creating the context block and subsequently when locating the associated information.

·    FsRtlRemovePerStreamContext – this call is used by the file system filter driver to remove its per-stream context information associated with the stream context of the file object provided to this call.  Typically, a file system filter driver would allow the underlying file system to discard this information when the stream data structures are uninitialized.  Thus, this call is normally only used when the filter determines that it no longer wishes to associate this state with the file (the documentation in the IFS Kit provides more details about the use of this call).

·    FsRtlSupportsPerStreamContexts – this call is used by a file system filter driver to determine if the given file object supports context tracking.  If the file object does not support such tracking, a file system may track context independently using some sort of lookup table mechanism (similar to techniques used with Windows 2000).

·    FsRtlTeardownPerStreamContexts – this call is used by a file system (and not the file system filter driver) to delete all per-stream file system filter driver contexts.  The operating system will call the “deallocation” routine provided by each filter that has a registered context.

These new callback mechanisms are substantially more efficient than a table-driven lookup mechanism as they can rely directly upon data associated with the given file control block.

Another new API deals with the handling of six fast I/O functions related to virtual memory locking.  To intercept these calls, a file system filter driver calls FsRtlRegisterFile SystemFilterCallbacks.  In turn, the six routines FsRtlAcquireFileExclusive, FsRtlReleaseFile, FsRtlAcquire FileForCcFlushEx, FsRtlReleaseFileForCcFlush, FsRtlAcquireFileForModWriteEx, and FsRtlRelease FileForModWrite will then call into a file system filter driver that has registered to handle these calls.  Further, a file system filter driver may be called before the file system has been called (or the “default” behavior has been executed if the file system does not implement the relevant fast I/O entry point) as well as afterwards.

Subsequent to registration, a file system filter driver will be called at its registered callback functions with information about the ongoing operation.  It is provided with information about the specific operation – the device object, the file object, and the operation-specific parameters.  A file system filter driver may then perform its processing and allow the operation to continue.

One potential use for this API is that it may be used to block user level mapping of a file.  This is accomplished by failing the call to PreAcquireForSectionSynchronization when the SyncType parameter is SyncTypeCreateSection.  Note that a file system filter driver should not fail any other calls to its PreAcquireForSectionSynchronization entry point as doing so will lead to incorrect system behavior.

We do not recommend that a file system filter driver block the creation of memory mapped files as this may materially impact the ability of applications that rely upon memory mapped file access to function properly.  While some applications have been written so that they can function in cases where memory mapped file services are not available, this is not the case for all functions.

Changes to Existing APIs

In addition to numerous new APIs, there are some changes to existing APIs that may have an impact on file systems and file system filter drivers.  Of course, the ultimate determination of this is the responsibility of the developer.

·    PsDereferencePrimaryToken – this has been converted from a macro (in the Windows 2000 IFS Kit) to a function (in the Windows XP IFS Kit).   Thus, if your file system or file system filter driver utilizes this function you will need to recompile to ensure you are using the correct implementation of this operation.

·    PsDereferenceImpersonationToken – this has been converted from a macro (in the Windows 2000 IFS Kit) to a function (in the Windows XP IFS Kit).  Thus, if your file system or file system filter driver utilizes this function you will need to recompile to ensure you are using the correct implementation of this operation.

·    CcMapData – the fourth parameter to this function has been converted from a BOOLEAN value (in the Windows 2000 IFS Kit) to an integer value (in the Windows XP IFS Kit).  The new function uses the value 1 (MAP_WAIT) to indicate that the call may block (wait).  This would be equivalent to setting the Wait parameter in Windows 2000 to be TRUE (which is defined as being 1).  In addition, the Windows XP routine also recognizes the MAP_NO_READ flag in this field.  If present, it will prevent the cache manager from faulting in the data within this region as part of this call.  (Note that it appears the documentation in the IFS Kit for this call is for the earlier version, not for the Windows XP implementation.  The changes are present in the header file).

·    Generic Table package – the generic table package (e.g. RtlInitializeGenericTable and associated functions) has been re-implemented utilizing AVL (balanced binary trees) rather than the earlier splay tree based implementation.  The legacy implementation is present in the OS binary image (in order to preserve binary compatibility where possible) but any compilation with the newest version of ntifs.h will utilize the new AVL table package.  While it appears to be compatible, it is worth being aware of this change if you are using the generic table package.

·    The definition of KeNumberProcessors has changed.  In previous versions, this value was referenced via a pointer to obtain the value in the kernel.  In Windows XP this value is now directly obtained.  Thus, any code that previously referenced this field was required to dereference it.  For Windows XP it should no longer dereference it.

·    The Ex work routines are now considered to be obsolete.  Drivers must either utilize the I/O Manager routines (IoQueueWorkItem, IoAllocateWorkItem, etc.) or they must implement their own work queue package.

For those file systems and file system filter drivers using these APIs, the responsible developer should review their usage of these to ensure that the new semantics and changes are properly reflected in their code base.

New Security APIs

The Windows XP release includes a rather substantial number of changes in the exposed security interface.  A number of these are used for exposing existing APIs for use in file systems and file system filter drivers.  Some additional changes are present that will simplify the development of file systems and file system filter drivers.  Given that the security changes justify a complete article in their own right, we shall touch briefly upon the new APIs as well as how they may affect a file system filter driver.

·    NtOpenThreadToken, NtOpenThreadToken Ex, ZwOpenThreadTokenEx – these calls allow a driver to open a token, registered to the specific thread for subsequent operations.  If the thread does not have a token, this call will fail.

·    NtOpenProcessToken, NtOpenProcessToken Ex, ZwOpenProcessTokenEx – these calls allow a driver to open a process token.  Since a process always has a token, this call should normally succeed, although it can fail for ancillary reasons (e.g., resource exhaustion, security check failures, etc.)

·    NtOpenJobObjectToken – this call allows a driver to open the token for a given job.  In general, this is of marginal use to driver developers as there is no exposed mechanism for obtaining a job handle, although it may be possible to combine user-mode functionality (to obtain the job handle) with driver functionality.

·    NtDuplicateToken – this call allows a driver to duplicate an existing token (of course, it requires appropriate security permission – TOKEN_DUPLICATE has been granted on the existing token handle).

·    NtFilterToken – this call allows a driver to create a restricted copy of an existing token.  It disables specific groups and privileges as well as allows associating new “restricted SIDs” with the token.

·    NtImpersonateAnonymousToken – this call allows a driver to bind a thread to the “anonymous user” token.  It restricts the actions of this thread to those that would be allowed by any unauthenticated user of the system.

·    NtQueryInformationToken, ZwQuery InformationToken – these calls are used to obtain information about the given token.  For example, a file system filter driver might use these calls to obtain the SID of a given thread or process for reporting or tracking purposes.

·    NtSetInformationToken, ZwSetInformation Token – these calls are used to modify information maintained by the system about the given token.

·    NtAdjustPrivilegesToken – this call is used by a driver in order to enable or disable privileges for the given token.

·    NtAdjustGroupsToken – this call is used by a driver in order to modify the list of group SIDs associated with a given token.

·    NtPrivilegeCheck – this call is used by a driver to validate that the list of privileges is present in the given token.  Typically, this would be used to implement a driver-specific security policy.

·    NtAccessCheckAndAuditAlarm, NtAccess CheckByTypeAndAuditAlarm, NtAccessCheck ByTypeResultListAndAuditAlarm, NtAccess CheckByTypeResultListAndAuditAlarmBy Handle – these calls are used to perform an access check and audit the results of that operation (if necessary) based upon the results of the operation.

·    NtOpenObjectAuditAlarm – this call is used by a driver that wishes to generate audit messages.  Typically, these would be tied into the security policy of the given driver (e.g., such auditing or alarm events would be triggered during a security check).

·    NtPrivilegeObjectAuditAlarm – this call is used by a driver that wishes to generate audit messages.  Typically, these would be tied into the security policy of the given driver where the security choice is based upon privilege checks.

·    NtCloseObjectAuditAlarm – this call is used by a driver that wishes to generate an audit message.  Typically, the driver would tie this into its internal security policy requiring that an audit (or alarm) be triggered upon closing the object in question.

·    NtDeleteObjectAuditAlarm – this call is used by a driver that wishes to generate an audit message.  Typically, the driver would tie this into its internal policy requiring that an audit (or alarm) be triggered upon deletion of an object.

·    NtPrivilegedServiceAuditAlarm – this call is used by the OS when a caller attempts to invoke privileged OS services.  In theory, a driver could utilize this to log comparable auditing information.

·    ZwQuerySecurityObject – this call is used by a driver to retrieve the security descriptor associated with a given object.  In the case of a file object, this call will trigger a call into the FSD to retrieve the security descriptor (via IRP_MJ_QUERY_ SECURITY).

·    ZwSetSecurityObject – this call is used by a driver to replace the security descriptor associated with a given object.  In the case of a file object, this call will trigger a call into the FSD to store the security descriptor (via IRP_MJ_SET_ SECURITY).

·    SeAuditingHardLinkEvents – this call is used by a file system driver to determine if the auditing policy requires auditing hard link creation events.

·    PsAssignImpersonationToken – this call is used by a driver to associate the given impersonation token with the specified thread.

·    PsDisableImpersonation – this call is used to stop impersonation by a given thread.  The impersonation token remains referenced (the returned value of this structure “owns” the reference) and it may be restored later within the driver.

·    PsRestoreImpersonation – this call is used to replace the existing impersonation of a thread (if any) with the saved impersonation (from an earlier call to PsDisableImpersonation).

·    SecLookupAccountName – this call is one instance of several new “security context” operations.  File system filter drivers may find this useful for retrieving the textual name associated with a given SID.

There are many other security related changes as well but in general are of limited interested to file systems or file system filter drivers.

Other Changes

There are numerous changes present in the IFS Kit that are beyond the scope of this article.  A few of interest:

·    ZwLoadDriver, ZwUnloadDriver – these two routines allow drivers to control the loading of other drivers in the OS.

·    ZwDeleteFile – it is now possible to initiate a file deletion from a driver.

·    ZwQueryDirectoryFile – this allows a file system filter driver to query the contents of a directory, using the handle-based Zw API.

·    Numerous changes to MDL handling within the OS itself.  This includes the support for 64-bit addresses within an MDL (required for the IA64 version of Windows XP) as well as more stringent semantics (with respect to mappings) and implementation.

·    New cache manager functions (CcMdlWrite Abort, CcSetLogHandleForFile, and CcGet DirtyPages).

·    New FSRTL_ADVANCED_FCB_HEADER structure that file systems should utilize to allow them to support file system filter contexts.

For those actively developing and maintaining file system filter drivers, it is important to keep in mind that even with this many changes, the vast majority of the underlying mechanism has not changed in any fundamental fashion.  Thus, file systems and file system filter drivers written for the Windows 2000 environment might require some changes, but a typical porting effort should be no more than a few days of work.