Driver Problems? Questions? Issues?
Put OSR's experience to work for you! Contact us for assistance with:
  • Creating the right design for your requirements
  • Reviewing your existing driver code
  • Analyzing driver reliability/performance issues
  • Custom training mixed with consulting and focused directly on your specific areas of interest/concern.
Check us out. OSR, the Windows driver experts.

On-Access, Transparent, Per-File Data Encryption:

OSR's File Encryption Solution Framework (FESF) provides all the infrastructure you need to build a transparent file encryption product REALLY FAST.

Super flexible policy determination and customization, all done in user-mode. Extensive starter/sample code provided.

Proven, robust, flexible. In use in multiple commercial products.

Currently available on Windows. FESF for Linux will ship in 2018.

For more info: https://www.osr.com/fesf

Go Back   OSR Online Lists > ntfsd
Welcome, Guest
You must login to post to this list
  Message 1 of 7  
26 Jan 13 11:56
Muhammad Umair
xxxxxx@gmail.com
Join Date: 29 Dec 2012
Posts To This List: 21
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

Okay so i was looking at the "Delete File System Minifilter Driver" sample code here for WDK 8: http://code.msdn.microsoft.com/windowshardware/Delete-File-System-b904651d/source code?fileId=51249&pathId=2034842209 The sample detects and reports deletions of files and streams using different techniques. -One of the techniques it uses is to flag a file/stream for which a FileDispositionInformation class has been set. For this purposes it uses the IRP_MJ_SET_INFORMATION major function. -Another technique is to check for files which have been flaged as "FILE_DELETE_ON_CLOSE" during creation. For that it uses the IRP_MJ_CREATE major function. -It also uses other mechanisms to flag file deletions during transactions, but anyhow, thats not relevant here. Finally it 'verifies' the flagged file/stream deletions by using the IRP_MJ_CLEANUP major function. Below is the registration structure: CONST FLT_OPERATION_REGISTRATION Callbacks[] = { { IRP_MJ_CREATE, 0, SfPreCreateCallback, SfPostCreateCallback }, { IRP_MJ_SET_INFORMATION, FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO, SfPreSetInfoCallback, SfPostSetInfoCallback }, { IRP_MJ_CLEANUP, 0, SfPreCleanupCallback, SfPostCleanupCallback }, { IRP_MJ_OPERATION_END } }; Now i have read what the difference between caching, non-caching, and paging I/Os is from the IFS FAQ, but i am not that clear on it. So far this is my understanding: caching i/o: i/o operations performed on memory/virtual memory non-caching i/o: where the cache is purposefully bypassed paging i/o: performed by the virtual memory system to bring data from storage device into the memory Now, my questions are, -why the filter has been setup to ignore PAGING IO in the IRP_MJ_SET_INFORMATION? -similarly, why has the PAGING IO not been ignored in the IRP_MJ_CREATE and IRP_MJ_SET_INFORMATION? -- Using Opera's revolutionary email client: http://www.opera.com/mail/
  Message 2 of 7  
26 Jan 13 21:35
ntfsd member 8
xxxxxx@osr.com
Join Date:
Posts To This List: 2736
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

The paging I/O operations in the set information path change the file sizes= . Those cannot matter for deletion detection. There are no paging I/O operations for create or cleanup. The four operations that support paging I/O operations are: read, write, qu= ery information and set information. Tony OSR
  Message 3 of 7  
26 Jan 13 23:08
Tim Mu
xxxxxx@163.com
Join Date: 24 Nov 2012
Posts To This List: 7
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

@Tony like paging read/write, do VM sponsor Paging I/O set information(for query information) ?
  Message 4 of 7  
27 Jan 13 00:50
ntfsd member 8
xxxxxx@osr.com
Join Date:
Posts To This List: 2736
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

<quote> like paging read/write, do VM sponsor Paging I/O set information(for query = information) ?=20 </quote> If by "sponsor" you mean "initiate" or "send" then yes. Mm needs to modify size information in certain circumstances, for example w= hen a section is first created (NtCreateSection takes an optional "maximum = size" (http://msdn.microsoft.com/en-us/library/windows/hardware/ff566428%28= v=3Dvs.85%29.aspx) that translates to allocation size of the file.=20 It also needs to query the size, such as when you create a section and don'= t specify the size - in that case, it is established from the file's size. = Note that there are other times when Mm (or Cc) may adjust file sizes as w= ell. A concern for Mm is the guarantee that it will never do extending wri= te operations on a file - so a file must be grown before you can get a pagi= ng write to it. Tony OSR
  Message 5 of 7  
27 Jan 13 12:18
Muhammad Umair
xxxxxx@gmail.com
Join Date: 29 Dec 2012
Posts To This List: 21
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

Thankyou so much for the clarification! Forgive my lack of knowledge, i need a little explanation of how paged i/o works. I understand that, some of the memory will not be available physically, instead it will be paged to a storage device. I also understand that when you try to read something that is paged, the system will bring in that page onto the physical memory then it will be read. In case of for example a write operation, I have read about the different I/O types, direct, buffered and niether. How does the paged I/O fit into the write operation? I mean when the write operation is buffered then maybe the allocated buffer has been paged? but in case of say direct I/O, there is no buffer, than how will paged I/O fit into that? or will the MDL be paged or something? Similarly, the Neither I/O i think only uses a virtual address, no MDL or buffer. Forgive my confusion but i am just trying to figure out how this all fits together. On Sun, 27 Jan 2013 07:37:14 +0500, Tony Mason <xxxxx@osr.com> wrote: > The paging I/O operations in the set information path change the file > sizes. Those cannot matter for deletion detection. > > There are no paging I/O operations for create or cleanup. > > The four operations that support paging I/O operations are: read, write, > query information and set information. > > Tony > OSR <...excess quoted lines suppressed...> -- Using Opera's mail client: http://www.opera.com/mail/
  Message 6 of 7  
27 Jan 13 14:14
ntfsd member 8
xxxxxx@osr.com
Join Date:
Posts To This List: 2736
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

I spent many hours discussing this in my file systems class, so it's diffic= ult to capture that in an e-mail on the discussion forum. The issue of "direct" versus "buffered" versus "neither" relate to the tran= sfer of information between two distinct address spaces (user mode versus = kernel mode) while the job of the memory manager is to control the mapping = of pages within an address space ("virtual address") to its corresponding d= ata, which includes physical memory. When a virtual-to-physical mapping is= not defined, the hardware cannot do the translation and invokes the OS - t= his is known as a "page fault" and happens when the virtual to physical map= ping is not defined (for the hardware) OR when the operation on the memory = is not consistent with the protection on the virtual page, such as a write = to a read-only (virtual) page. In Windows the Memory Manager is responsible for handling virtual memory. W= hen a page fault occurs the kernel receives a processor fault (0xE on x86 a= nd x64 CPUs) and control is transferred to a function in the kernel (e.g., = KiTrap0E). That OS function does some preliminary analysis of the page faul= t and then normally transfers control to the Memory Manager (MmAccessFault)= . The hardware only has its translation cache ("translation lookaside buffer"= ) and the page tables (shared between OS and CPU) to consult. The Memory M= anager maintains a number of other data structures that allow it to "fix up= " the virtual to physical mapping so that it can "satisfy" the page fault. = For a region of virtual memory that represents a file backed section, Mm k= eeps a pointer to a file object. It can then allocate a physical page (or = pages) and call the I/O Manager to "fill in" those pages with the correct d= ata (IoPageRead, which is actually in ntifs.h). The IRPs built in this cas= e will have the IRP_PAGING_IO bit set. While it tells a file system and fi= lter that this is a paging I/O, the I/O Manager does it because it needs to= know how to properly clean up the IRP when it is done. For example, the I= /O Manager does not use APCs to indicate completion of a paging I/O IRP - i= t directly sets the event object in the IRP, as the memory manager has neve= r allowed APCs during paging I/O (XP and earlier it was done by raising to = IRQL APC_LEVEL, S03 and beyond it's done by using a guarded mutex, which di= sables all APCs via a different mechanism that doesn't raise IRQL). So buffered/neither/direct have to do with how the OS transfers information= from the user address space into the OS address space: - Buffered allocates a kernel buffer and copies the data (it is "captured") - Direct builds an MDL that describes the virtual pages of the user data. = When you "lock" those pages, it means that the Virtual-to-Physical translat= ion is fixed. Those physical pages cannot be used for anything else by the= OS as long as the buffer is locked. The kernel address is then constructe= d by using the locked MDL description to create a second virtual-to-physica= l mapping to those pages ("MmGetSystemAddressForMdlSafe" which calls MmMapL= ockedPages when the mapping hasn't been set in the MDL yet, or uses the map= ping indicated in the MDL). The advantage of this approach is that there's= no data copy. The disadvantage of this approach is that the contents of t= he buffer are shared with user mode and thus are not captured. But the ker= nel address is valid and remains so until the mapping is torn down. - Neither hands the user's buffer address to the OS. It may or may not be = valid and it may be valid at the start of an operation and go invalid by th= e end of the operation. I hope this helps clarify these concepts in your mind. VM is amazingly com= plicated and the file systems in Windows and the VM system are very intertw= ined with one another. Tony OSR -----Original Message----- From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] = On Behalf Of Muhammad Umair Sent: Sunday, January 27, 2013 9:17 AM To: ntfsd redirect Subject: Re:[ntfsd] [FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is th= is flag used here? (minifilter) Thankyou so much for the clarification! Forgive my lack of knowledge, i need a little explanation of how paged i/o = works. I understand that, some of the memory will not be available physical= ly, instead it will be paged to a storage device. I also understand that wh= en you try to read something that is paged, the system will bring in that p= age onto the physical memory then it will be read. In case of for example a write operation, I have read about the different I= /O types, direct, buffered and niether. How does the paged I/O fit into the= write operation? I mean when the write operation is buffered then maybe th= e allocated buffer has been paged? but in case of say direct I/O, there is = no buffer, than how will paged I/O fit into that? or will the MDL be paged = or something? Similarly, the Neither I/O i think only uses a virtual addres= s, no MDL or buffer. Forgive my confusion but i am just trying to figure ou= t how this all fits together. On Sun, 27 Jan 2013 07:37:14 +0500, Tony Mason <xxxxx@osr.com> wrote: > The paging I/O operations in the set information path change the file=20 > sizes. Those cannot matter for deletion detection. > > There are no paging I/O operations for create or cleanup. > > The four operations that support paging I/O operations are: read,=20 > write, query information and set information. > > Tony > OSR <...excess quoted lines suppressed...> -- Using Opera's mail client: http://www.opera.com/mail/ --- NTFSD is sponsored by OSR OSR is hiring!! Info at http://www.osr.com/careers For our schedule of debugging and file system seminars visit:=20 http://www.osr.com/seminars To unsubscribe, visit the List Server section of OSR Online at http://www.o= sronline.com/page.cfm?name=3DListServer
  Message 7 of 7  
27 Jan 13 16:20
Muhammad Umair
xxxxxx@gmail.com
Join Date: 29 Dec 2012
Posts To This List: 21
[FLTFL_OPERATION_REGISTRATION_SKIP_PAGING_IO] Why is this flag used here? (minifilter)

Thankyou so much for the detailed response! ...that actually does clear up alot! ...windows internals are turning out to be pretty interesting even though this is the first time i've had an encounter with em as a student. On Mon, 28 Jan 2013 00:15:58 +0500, Tony Mason <xxxxx@osr.com> wrote: > I spent many hours discussing this in my file systems class, so it's > difficult to capture that in an e-mail on the discussion forum. > > The issue of "direct" versus "buffered" versus "neither" relate to the > transfer of information between two distinct address spaces (user mode > versus kernel mode) while the job of the memory manager is to control > the mapping of pages within an address space ("virtual address") to its > corresponding data, which includes physical memory. When a > virtual-to-physical mapping is not defined, the hardware cannot do the > translation and invokes the OS - this is known as a "page fault" and <...excess quoted lines suppressed...> -- Using Opera's mail client: http://www.opera.com/mail/
Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You must login to OSR Online AND be a member of the ntfsd list to be able to post.

All times are GMT -5. The time now is 04:21.


Copyright ©2015, OSR Open Systems Resources, Inc.
Based on vBulletin Copyright ©2000 - 2005, Jelsoft Enterprises Ltd.
Modified under license