Driver Problems? Questions? Issues?
Put OSR's experience to work for you! Contact us for assistance with:
  • Creating the right design for your requirements
  • Reviewing your existing driver code
  • Analyzing driver reliability/performance issues
  • Custom training mixed with consulting and focused directly on your specific areas of interest/concern.
Check us out. OSR, the Windows driver experts.

On-Access, Transparent, Per-File Data Encryption:

OSR's File Encryption Solution Framework (FESF) provides all the infrastructure you need to build a transparent file encryption product REALLY FAST.

Super flexible policy determination and customization, all done in user-mode. Extensive starter/sample code provided.

Proven, robust, flexible. In use in multiple commercial products.

Currently available on Windows. FESF for Linux will ship in 2018.

For more info: https://www.osr.com/fesf

Go Back   OSR Online Lists > ntfsd
Welcome, Guest
You must login to post to this list
  Message 1 of 11  
05 Jul 17 10:46
Mickey H
xxxxxx@gmail.com
Join Date: 23 Jul 2017
Posts To This List: 19
Hide minifilter write operations

Hi, I am pretty new to minifilter drivers so please forgive me if I am missing anything obvious. I am trying to write a minifilter driver which handles my stub files. My stub files are empty files with the original logical size (sparse file with a hole). When some application tries to access a stub, the minifilter driver intercepts the pre-read operation and my service starts to download the real file. Once file is downloaded, the pre-read can continue. My problem is that "Windows defender" (MsMpEng.exe) detects my service writing to the file, and for some reason rewrites to this file when the service closes it. This write operation by windows defender is problematic for my application. I tried to avoid it by making the write call from the service to the minifilter directly (through filter communication port), in the minifilter I use "FltWriteFile", but that didn't help. Windows defender still detects my minifilter write operations even though my minifilter's altitude is lower than "WdFilter". Any idea why windows defender re-writes the entire file? Any idea how I can hide my write operations from windows defender? Thanks, Mickey
  Message 2 of 11  
05 Jul 17 12:41
Scott Noone
xxxxxx@osr.com
Join Date: 10 Jul 2002
Posts To This List: 942
List Moderator
Hide minifilter write operations

Can you print the call stack for the duplicate write that you're seeing? Also, how do you fill the read buffer once the file population is complete? Do you really just pend the callback data and send it later or are you doing something with the original read buffer? -scott OSR @OSRDrivers wrote in message news:102730@ntfsd... Hi, I am pretty new to minifilter drivers so please forgive me if I am missing anything obvious. I am trying to write a minifilter driver which handles my stub files. My stub files are empty files with the original logical size (sparse file with a hole). When some application tries to access a stub, the minifilter driver intercepts the pre-read operation and my service starts to download the real file. Once file is downloaded, the pre-read can continue. My problem is that "Windows defender" (MsMpEng.exe) detects my service writing to the file, and for some reason rewrites to this file when the service closes it. This write operation by windows defender is problematic for my application. I tried to avoid it by making the write call from the service to the minifilter directly (through filter communication port), in the minifilter I use "FltWriteFile", but that didn't help. Windows defender still detects my minifilter write operations even though my minifilter's altitude is lower than "WdFilter". Any idea why windows defender re-writes the entire file? Any idea how I can hide my write operations from windows defender? Thanks, Mickey
  Message 3 of 11  
05 Jul 17 15:58
Mickey H
xxxxxx@gmail.com
Join Date: 23 Jul 2017
Posts To This List: 19
Hide minifilter write operations

Thank you for your quick reply. I believe that Windows Defender behavior is not related to my changes. I wrote some utility that overwrites a file with a specific string. I ran it while my filter was not installed on the machine (and my service is disabled). I see in Process Monitor that windows defender rewrites my file after is gets closed: 10:39:53.1444234 PM xappend.exe 5524 CreateFile E:\Downloads\Mickey.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened 10:39:53.1444838 PM xappend.exe 5524 WriteFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 15, Priority: Normal 10:39:53.1445793 PM xappend.exe 5524 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1573086 PM xappend.exe 5524 CloseFile E:\Downloads\Mickey.jpg SUCCESS 10:39:53.1588688 PM MsMpEng.exe 1112 CreateFileMapping E:\Downloads\Mickey.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 10:39:53.1588867 PM MsMpEng.exe 1112 QueryStandardInformationFile E:\Downloads\Mickey.jpg SUCCESS AllocationSize: 2,424,832, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 10:39:53.1589683 PM MsMpEng.exe 1112 WriteFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1594533 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1600757 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 2,355,200, Length: 5,442, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1629744 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 61,440, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal Regarding my read operation, I just pend the operation and resume it when the file is populated.
  Message 4 of 11  
05 Jul 17 17:15
Scott Noone
xxxxxx@osr.com
Join Date: 10 Jul 2002
Posts To This List: 942
List Moderator
Hide minifilter write operations

You're seeing a flush of cached data. Check the call stack for the write from the context of the MsMpEng process, probably a CcFlushCache from the file system. This wouldn't be Defender writing the file, it's just that its data scan is causing the file system to flush the cache. -scott OSR @OSRDrivers wrote in message news:102733@ntfsd... Thank you for your quick reply. I believe that Windows Defender behavior is not related to my changes. I wrote some utility that overwrites a file with a specific string. I ran it while my filter was not installed on the machine (and my service is disabled). I see in Process Monitor that windows defender rewrites my file after is gets closed: 10:39:53.1444234 PM xappend.exe 5524 CreateFile E:\Downloads\Mickey.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened 10:39:53.1444838 PM xappend.exe 5524 WriteFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 15, Priority: Normal 10:39:53.1445793 PM xappend.exe 5524 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1573086 PM xappend.exe 5524 CloseFile E:\Downloads\Mickey.jpg SUCCESS 10:39:53.1588688 PM MsMpEng.exe 1112 CreateFileMapping E:\Downloads\Mickey.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 10:39:53.1588867 PM MsMpEng.exe 1112 QueryStandardInformationFile E:\Downloads\Mickey.jpg SUCCESS AllocationSize: 2,424,832, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 10:39:53.1589683 PM MsMpEng.exe 1112 WriteFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1594533 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1600757 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 2,355,200, Length: 5,442, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal 10:39:53.1629744 PM MsMpEng.exe 1112 ReadFile E:\Downloads\Mickey.jpg SUCCESS Offset: 61,440, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal Regarding my read operation, I just pend the operation and resume it when the file is populated.
  Message 5 of 11  
06 Jul 17 17:32
Mickey H
xxxxxx@gmail.com
Join Date: 23 Jul 2017
Posts To This List: 19
Hide minifilter write operations

You are right, thank you for your help. I did some reading in order to understand how the cache manager works. In order to bypass it, I decided to use the FILE_FLAG_NO_BUFFERING flag when I call CreateFile in my service. According to the documentation, when using this flag the cache manager should ignore my service write operations. Now I do see that my service initiates non-cached IO requests, but I still have a problem. The cache manager still writes the last 4K of the file after my service closes the file. I suspect that this write operation is because of my service setting the end of file size (FILE_FLAG_NO_BUFFERING requires writing aligned buffers). I use SetFileInformationByHandle in order to set the actual size. This is what I see in Process Monitor: 11:40:34.5217380 PM xSrv.exe 3576 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: No Buffering, Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: Read, AllocationSize: n/a, OpenResult: Opened 11:40:34.5226546 PM xSrv.exe 3576 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 1,048,576, I/O Flags: Non-cached, Priority: Normal 11:40:34.5362022 PM xSrv.exe 3576 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 1,048,576, Length: 1,048,576, I/O Flags: Non-cached, Priority: Normal 11:40:34.5396911 PM xSrv.exe 3576 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,097,152, Length: 266,240, I/O Flags: Non-cached, Priority: Normal 11:40:34.5432151 PM xSrv.exe 3576 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfF ile: 2,360,642 11:40:34.5433648 PM xSrv.exe 3576 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Priority: Normal 11:40:34.5443753 PM xSrv.exe 3576 CloseFile E:\Downloads\test.jpg SUCCESS 11:40:34.7266040 PM DllHost.exe 4224 CloseFile E:\Downloads\test.jpg SUCCESS 11:40:34.7502330 PM MsMpEng.exe 1756 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 11:40:34.7502548 PM MsMpEng.exe 1756 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS Al locationSize: 2,424,832, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 11:40:34.7503037 PM MsMpEng.exe 1756 WriteFile E:\Downloads\test.jpg CANCELLED Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal Please note that the last write gets canceled because it causes a bug on my side. The call stack for the last write: 0 FLTMGR.SYS FltDecodeParameters + 0x195a 0xfffff80d605146ca C:\Windows\System32\drivers\FLTMGR.SYS 1 FLTMGR.SYS FltDecodeParameters + 0x1508 0xfffff80d60514278 C:\Windows\System32\drivers\FLTMGR.SYS 2 FLTMGR.SYS FltDecodeParameters + 0x616 0xfffff80d60513386 C:\Windows\System32\drivers\FLTMGR.SYS 3 FLTMGR.SYS FltDecodeParameters + 0x3be 0xfffff80d6051312e C:\Windows\System32\drivers\FLTMGR.SYS 4 ntoskrnl.exe IoSynchronousPageWrite + 0x15c 0xfffff8029a564afc C:\Windows\system32\ntoskrnl.exe 5 ntoskrnl.exe PsGetProcessWin32Process + 0x44a 0xfffff8029a576c9a C:\Windows\system32\ntoskrnl.exe 6 ntoskrnl.exe PsChargeProcessNonPagedPoolQuota + 0xcb2 0xfffff8029a4d8112 C:\Windows\system32\ntoskrnl.exe 7 ntoskrnl.exe KeAcquireInStackQueuedSpinLock + 0x8003 0xfffff8029a512fd3 C:\Windows\system32\ntoskrnl.exe 8 ntoskrnl.exe CcSetParallelFlushFile + 0x440 0xfffff8029a4f971c C:\Windows\system32\ntoskrnl.exe 9 ntoskrnl.exe CcFlushCache + 0x16 0xfffff8029a51aad6 C:\Windows\system32\ntoskrnl.exe 10 ntoskrnl.exe FsRtlCreateSectionForDataScan + 0x297 0xfffff8029a4a4a87 C:\Windows\system32\ntoskrnl.exe 11 ntoskrnl.exe FsRtlCreateSectionForDataScan + 0x11f 0xfffff8029a4a490f C:\Windows\system32\ntoskrnl.exe 12 FLTMGR.SYS FltCreateSectionForDataScan + 0xec 0xfffff80d6054761c C:\Windows\System32\drivers\FLTMGR.SYS 13 WdFilter.sys WdFilter.sys + 0x19578 0xfffff80d5fd19578 C:\Windows\system32\drivers\WdFilter.sys 14 WdFilter.sys WdFilter.sys + 0x19030 0xfffff80d5fd19030 C:\Windows\system32\drivers\WdFilter.sys 15 FLTMGR.SYS FltDeleteStreamHandleContext + 0x146 0xfffff80d60518fb6 C:\Windows\System32\drivers\FLTMGR.SYS 16 FLTMGR.SYS FltRemoveOpenReparseEntry + 0x33f 0xfffff80d6054345f C:\Windows\System32\drivers\FLTMGR.SYS 17 FLTMGR.SYS FltDecodeParameters + 0x409 0xfffff80d60513179 C:\Windows\System32\drivers\FLTMGR.SYS 18 ntoskrnl.exe NtQueryInformationFile + 0xc20 0xfffff8029a8a8db0 C:\Windows\system32\ntoskrnl.exe Any idea why the last 4k are written by the cache manager?
  Message 6 of 11  
06 Jul 17 23:43
Slava Imameev
xxxxxx@hotmail.com
Join Date: 13 Sep 2013
Posts To This List: 264
Hide minifilter write operations

<QUOTE> According to the documentation, when using this flag the cache manager should ignore my service write operations. </QUOTE> A file system driver might not call the Cache Manager to perform read-write. The Cache Manager doesn't care about this flag. <QUOTE> Any idea why the last 4k are written by the cache manager? </QUOTE> The file is cached. File system might ignore non-cached flags for its own reasons ( and NTFS does this for some files ) or use cache to maintain file coherence in presence of memory mapped file sections.
  Message 7 of 11  
07 Jul 17 08:12
Mickey H
xxxxxx@gmail.com
Join Date: 23 Jul 2017
Posts To This List: 19
Hide minifilter write operations

According to "Windows Internals" (Chapter 11, Cache Manager): "Keep in mind that the cache manager is involved in file I/O only when a file is opened without the FILE_FLAG_NO_BUFFERING flag and then read from or written to using the Windows I/O functions (for example, using the Windows ReadFile and WriteFile functions)." I wrote a simple utility that writes zeros to an existing file. It writes zeros in 4K buffers, and at the end it sets the size to be the original size of the file. The first version of the utility opens the file without any special flags, the second version (xzero_no_buff) opens the file with the flag FILE_FLAG_NO_BUFFERING. * First version Process Monitor output: 2:51:16.3944250 PM xzero.exe 6412 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened 2:51:16.3944703 PM xzero.exe 6412 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS Allo cationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 2:51:16.3944834 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 4,096, Priority: Normal 2:51:16.3945298 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 4,096, Length: 4,096 ... 2:51:16.4019936 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,355,200, Length: 4,096 2:51:16.4020009 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096 2:51:16.4020285 PM xzero.exe 6412 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOf File: 2,360,642 2:51:16.4090436 PM xzero.exe 6412 CloseFile E:\Downloads\test.jpg SUCCESS 2:51:16.4094012 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 2:51:16.4094244 PM MsMpEng.exe 372 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS All ocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 2:51:17.0794587 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 2:51:17.0821091 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 2:51:18.3802095 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 1,048,576, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 2:51:18.3829431 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 2:51:19.7413443 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,097,152, Length: 266,240, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 2:51:19.7424212 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642 2:51:19.7424416 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther * Second version Process Monitor output: 3:04:27.8110033 PM xzero_no_buff.exe 7016 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: No Buffering, Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened 3:04:27.8207308 PM xzero_no_buff.exe 7016 QueryStandardInformationFile E:\Downloads\test.jpg SUCC ESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 3:04:27.8207705 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Priority: Normal 3:04:27.8211333 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 4,096, Length: 4,096, I/O Flags: Non-cached, Priority: Normal ... 3:04:27.9064179 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Priority: Normal 3:04:27.9065758 PM xzero_no_buff.exe 7016 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCE SS EndOfFile: 2,360,642 3:04:27.9066143 PM xzero_no_buff.exe 7016 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Priority: Normal 3:04:27.9086864 PM xzero_no_buff.exe 7016 CloseFile E:\Downloads\test.jpg SUCCESS 3:04:27.9090498 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 3:04:27.9090725 PM MsMpEng.exe 372 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS All ocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 3:04:27.9091353 PM MsMpEng.exe 372 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal ... 3:04:30.8326217 PM System 4 CreateFileMapping C:\test\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 3:04:30.8445425 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,359,296 3:04:30.8445975 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 3:04:32.1532605 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 3:04:32.1566545 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642 3:04:32.1567095 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther As you can see, in the first version the entire file is written by the cache manager, while in the second version only the last 4KB frame is written. I suspect that the last frame is being written because I set the file size. How can I make the SetEndOfFileInformationFile a non-cached operation?
  Message 8 of 11  
07 Jul 17 09:27
Scott Noone
xxxxxx@osr.com
Join Date: 10 Jul 2002
Posts To This List: 942
List Moderator
Hide minifilter write operations

You don't appear to have deleted the file between the two runs, note that in your second version the file already has a non-zero AllocationSize and EndOfFile. So it would appear that you're mixing cached I/O and non-cached I/O to the same file. Try writing a new file out using non-cached I/O and see if you get different results. -scott OSR @OSRDrivers wrote in message news:102743@ntfsd... According to "Windows Internals" (Chapter 11, Cache Manager): "Keep in mind that the cache manager is involved in file I/O only when a file is opened without the FILE_FLAG_NO_BUFFERING flag and then read from or written to using the Windows I/O functions (for example, using the Windows ReadFile and WriteFile functions)." I wrote a simple utility that writes zeros to an existing file. It writes zeros in 4K buffers, and at the end it sets the size to be the original size of the file. The first version of the utility opens the file without any special flags, the second version (xzero_no_buff) opens the file with the flag FILE_FLAG_NO_BUFFERING. * First version Process Monitor output: 2:51:16.3944250 PM xzero.exe 6412 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened 2:51:16.3944703 PM xzero.exe 6412 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 2:51:16.3944834 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 4,096, Priority: Normal 2:51:16.3945298 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 4,096, Length: 4,096 ... 2:51:16.4019936 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,355,200, Length: 4,096 2:51:16.4020009 PM xzero.exe 6412 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096 2:51:16.4020285 PM xzero.exe 6412 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642 2:51:16.4090436 PM xzero.exe 6412 CloseFile E:\Downloads\test.jpg SUCCESS 2:51:16.4094012 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 2:51:16.4094244 PM MsMpEng.exe 372 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 2:51:17.0794587 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 2:51:17.0821091 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 2:51:18.3802095 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 1,048,576, Length: 1,048,576, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 2:51:18.3829431 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 2:51:19.7413443 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,097,152, Length: 266,240, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 2:51:19.7424212 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642 2:51:19.7424416 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther * Second version Process Monitor output: 3:04:27.8110033 PM xzero_no_buff.exe 7016 CreateFile E:\Downloads\test.jpg SUCCESS Desired Access: Generic Write, Read Attributes, Disposition: Open, Options: No Buffering, Synchronous IO Non-Alert, Non-Directory File, Attributes: N, ShareMode: None, AllocationSize: n/a, OpenResult: Opened 3:04:27.8207308 PM xzero_no_buff.exe 7016 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 3:04:27.8207705 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 4,096, I/O Flags: Non-cached, Priority: Normal 3:04:27.8211333 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 4,096, Length: 4,096, I/O Flags: Non-cached, Priority: Normal ... 3:04:27.9064179 PM xzero_no_buff.exe 7016 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Priority: Normal 3:04:27.9065758 PM xzero_no_buff.exe 7016 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642 3:04:27.9066143 PM xzero_no_buff.exe 7016 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Priority: Normal 3:04:27.9086864 PM xzero_no_buff.exe 7016 CloseFile E:\Downloads\test.jpg SUCCESS 3:04:27.9090498 PM MsMpEng.exe 372 CreateFileMapping E:\Downloads\test.jpg FILE LOCKED WITH WRITERS SyncType: SyncTypeCreateSection, PageProtection: PAGE_EXECUTE_READ|PAGE_NOCACHE 3:04:27.9090725 PM MsMpEng.exe 372 QueryStandardInformationFile E:\Downloads\test.jpg SUCCESS AllocationSize: 2,363,392, EndOfFile: 2,360,642, NumberOfLinks: 1, DeletePending: False, Directory: False 3:04:27.9091353 PM MsMpEng.exe 372 ReadFile E:\Downloads\test.jpg SUCCESS Offset: 0, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal ... 3:04:30.8326217 PM System 4 CreateFileMapping C:\test\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 3:04:30.8445425 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,359,296 3:04:30.8445975 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther 3:04:32.1532605 PM System 4 WriteFile E:\Downloads\test.jpg SUCCESS Offset: 2,359,296, Length: 4,096, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Very Low 3:04:32.1566545 PM System 4 SetEndOfFileInformationFile E:\Downloads\test.jpg SUCCESS EndOfFile: 2,360,642 3:04:32.1567095 PM System 4 CreateFileMapping E:\Downloads\test.jpg SUCCESS SyncType: SyncTypeOther As you can see, in the first version the entire file is written by the cache manager, while in the second version only the last 4KB frame is written. I suspect that the last frame is being written because I set the file size. How can I make the SetEndOfFileInformationFile a non-cached operation?
  Message 9 of 11  
07 Jul 17 15:07
Slava Imameev
xxxxxx@hotmail.com
Join Date: 13 Sep 2013
Posts To This List: 264
Hide minifilter write operations

<QUOTE> "Keep in mind that the cache manager is involved in file I/O only when a file is opened without the FILE_FLAG_NO_BUFFERING flag and then read from or written to using the Windows I/O functions (for example, using the Windows ReadFile and WriteFile functions)." </QUOTE> This description is not complete. It doesn't tell that the Cache Manager is not involved if the flag is set. A file system driver makes a decision on file caching and can ignore FILE_NO_INTERMEDIATE_BUFFERING. That is why I advocate for opening/sharing Windows kernel sources. Natural languages are not precise. <QUOTE> How can I make the SetEndOfFileInformationFile a non-cached operation? </QUOTE> You can't make this for all possible file system driver implementations. A file system driver might decide to initialize a cache for non-paging End-Of-File operation. You can find this in FastFAT sources when a cache is initialized if a data section for a mapped file exists. Look at FatSetEndOfFileInfo .
  Message 10 of 11  
07 Jul 17 18:10
Mickey H
xxxxxx@gmail.com
Join Date: 23 Jul 2017
Posts To This List: 19
Hide minifilter write operations

My utility writes zeros to an existing jpg file. Between the tests I replace the zeroed file from the first run with the original file (This is why in both runs I have non-zero allocation size). The utility writes 4KB frames and at the end it sets the file size as the original size (using SetFileInformationByHandle). If the utility doesn't set the size at the end (only writes 4KB frames and closes the file), I do NOT see the last frame being written by the Cache Manager. I repeated the test several times and never saw the last frame written in this version of the utility.
  Message 11 of 11  
11 Jul 17 15:54
Scott Noone
xxxxxx@osr.com
Join Date: 10 Jul 2002
Posts To This List: 942
List Moderator
Hide minifilter write operations

How do you replace the file? Do you dismount the volume between test runs? Also, please post the call stack for the write at the end. -scott OSR @OSRDrivers wrote in message news:102747@ntfsd... My utility writes zeros to an existing jpg file. Between the tests I replace the zeroed file from the first run with the original file (This is why in both runs I have non-zero allocation size). The utility writes 4KB frames and at the end it sets the file size as the original size (using SetFileInformationByHandle). If the utility doesn't set the size at the end (only writes 4KB frames and closes the file), I do NOT see the last frame being written by the Cache Manager. I repeated the test several times and never saw the last frame written in this version of the utility.
Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You must login to OSR Online AND be a member of the ntfsd list to be able to post.

All times are GMT -5. The time now is 08:12.


Copyright ©2015, OSR Open Systems Resources, Inc.
Based on vBulletin Copyright ©2000 - 2005, Jelsoft Enterprises Ltd.
Modified under license