Shadow file objects questions (2)

Hi,

After some time I want to continue this topic, https://www.osronline.com/showthread.cfm?link=269300, to get better knowledge of SFO internals.

As Scott said, after calling FltCreateFileEx, I must fill the FsContext2 structure with a CCB that references the shadow file object.

a) Is there an api to create the CCB object or must create it manually?

b) Can you give me a brief on how to do next, i.e., when I recieve an IRP_MJ_READ request? Scott said something about switching context objects. Does it mean I have to switch pointers and, on “Post” callback, restore them?

c) If yes, is there a problem if file objects belongs to different volumes?

d) Is there any difference using this method than creating a context (FltAllocateContext) for the file object, storing the file object obtained by the call to FltCreateFileEx and later use it, for e.g., by calling FltReadFile on IRP_MJ_READ?

Regards,
Mauro.

a) It’s an opaque blob of memory to anyone outside of you. See the FAT
source code …

b) Scott is referring to updating the target file object in the callback
data, if you are in a mini-filter.

c) Yes, it does matter if they are on different volumes, you would also
need to change the flt_instance pointer and be aware of cross-volume
requests which may not work such as rename.

d) Yes, you need to deal with the locking model. For example, MM can
call into the stack to pre-aquire a lock on a file object and you need
to capture this to ensure that the lock is acquired on the correct file
object, the one you would swap out in the IO pathways. It can definitely
get tricky but you are getting into layered file systems so you need to
understand all the semantics of how this works.

Pete


Kernel Drivers
Windows File System and Device Driver Consulting
www.KernelDrivers.com http:</http:>
866.263.9295

------ Original Message ------
From: xxxxx@mauroleggieri.com.ar
To: “Windows File Systems Devs Interest List”
Sent: 11/30/2015 5:24:57 AM
Subject: [ntfsd] Shadow file objects questions (2)

>Hi,
>
>After some time I want to continue this topic,
>https://www.osronline.com/showthread.cfm?link=269300, to get better
>knowledge of SFO internals.
>
>As Scott said, after calling FltCreateFileEx, I must fill the
>FsContext2 structure with a CCB that references the shadow file object.
>
>a) Is there an api to create the CCB object or must create it manually?
>
>b) Can you give me a brief on how to do next, i.e., when I recieve an
>IRP_MJ_READ request? Scott said something about switching context
>objects. Does it mean I have to switch pointers and, on “Post”
>callback, restore them?
>
>c) If yes, is there a problem if file objects belongs to different
>volumes?
>
>d) Is there any difference using this method than creating a context
>(FltAllocateContext) for the file object, storing the file object
>obtained by the call to FltCreateFileEx and later use it, for e.g., by
>calling FltReadFile on IRP_MJ_READ?
>
>Regards,
>Mauro.
>
>
>—
>NTFSD is sponsored by OSR
>
>OSR is hiring!! Info at http://www.osr.com/careers
>
>For our schedule of debugging and file system seminars visit:
>http://www.osr.com/seminars
>
>To unsubscribe, visit the List Server section of OSR Online at
>http://www.osronline.com/page.cfm?name=ListServer

> c) If yes, is there a problem if file objects belongs to different volumes?

Yes.

If you use dumb reparse to redirect Vol1:File1 -> Vol2:File2, then you have issues with MoveFile path.

If opens Vol1:File1 (actually Vol2:File2 is opened), and then compares the volumes of Vol2:File2 and of Vol1:DstDir. This comparison is done in IO itself before getting to the FS stack, so the filters cannot handle it.

Properly implemented shadow file objects (FSD over FSD) solve this.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

Thanks Peter and Maxim.

b) Scott is referring to updating the target file object in the callback
data, if you are in a mini-filter.

So the whole concept is: when a request is received, I change the target file object with my FsContext2 value (and filter instance), continue with PreOp with callback, then on PostOp, restore the values and viol??

d) Yes, you need to deal with the locking model. For example, MM can
call into the stack to pre-aquire a lock on a file object and you need
to capture this to ensure that the lock is acquired on the correct file
object, the one you would swap out in the IO pathways. It can definitely
get tricky but you are getting into layered file systems so you need to
understand all the semantics of how this works.

Let’s see if I understand well. I receive a request, swap pointers, before PostOp a pre-acquire lock request appears but the target will be the shadow file object instead of the original file. If this is correct, why I should touch them again if the real target is already set? Or you mean, I must check and avoid swapping in this case because it was done in the original request?

Regards.

When a request is received, it’s targeted at YOUR file object. But since you want the operation to be performed against the actual target (as represented by the shadow file object) you must change the parameters to use that file object. This might sound simple but you’d be surprised at the complexity. Think about relative open operations (for example) or rename operations (which have multiple file objects). The concept is the same, regardless of whether or not this is a legacy filter (where you are setting up the parameters for the IO_STACK_LOCATION for the next driver) or a mini-filter (where you are modifying the callback data structure so the filter manager can set up the parameters for the IO_STACK_LOCATION for the next driver).

[quote]

d) Yes, you need to deal with the locking model. For example, MM can
call into the stack to pre-aquire a lock on a file object and you need
to capture this to ensure that the lock is acquired on the correct
file object, the one you would swap out in the IO pathways. It can
definitely get tricky but you are getting into layered file systems so
you need to understand all the semantics of how this works.

Let’s see if I understand well. I receive a request, swap pointers, before PostOp a pre-acquire lock request appears but the target will be the shadow file object instead of the original file. If this is correct, why I should touch them again if the real target is already set? Or you mean, I must check and avoid swapping in this case because it was done in the original request?

[quote]

The big reason to use the shadow file object model is because YOU own the file object, YOU own the cache and YOU control the interactions with Mm/Cc. So when you call CcInitializeCacheMap and pass it the original file object, it gets initialized. When Mm wants to perform some operation, it’s going to call and your filter will be invoked to lock the file against some sort of change (there are locks for section set-up/creation, which prevents size changes, there are locks for paging write operations, which prevents truncation below a high water mark, and there are lock for paging write operations initiated by the cache manager’s lazy writer, which prevents file state from changing from beneath the cache manager).

YOU must handle this locking.

Then you have to figure out how this translated (or if it translated) to locking of the lower file system (which must be done using the shadow file object).

The Original File Object has YOUR state. The Shadow File Object has the underlying FSD’s state. You manage your state using Cc/Mm/FsRtl. You manage the SFO’s state by invoking operations via the Filter Manager API, either in the context of your own call, or in the context of an operation you are processing.

And if you do this right you can stack this (because if two layers are good, three *must* be better!)

Fair warning: these are definitely the most difficult types of filters to write and get working properly.

Tony
OSR

Hi Tony. Thank you for the big explanation. I knew rename operations are a special cases but no idea about relative opens.

Tomorrow I will review your post with a fresh mind to understand it better.

And yes, I know I’m trying to swim in the ocean but like the challenge. :slight_smile:

Regards,
Mauro.

Hi again,

A couple of questions to close this topic (at least for now).

I was seeing the RelatedFileObject field. Let’s say the original file object points to file A and my SFO points to file B.

Later, in some other operation, can I receive a file open relative to file B?

On such case, the correct is to handle it as relative to the original file A taking into account also if the original file A was in another volume, right?

Regards,
Mauro.