Feasibility of FUSE for Windows

Hi all,

I’m currently a grad student taking an OS class at U Wisconsin, and I was fishing around for a project. One of the ideas I came up with was to create a Windows port of FUSE. (If you’re not familiar with it, I’ll provide a summary at the end of this message.) On its face it didn’t seem like it would be a rather possible project, but after doing some reading it seems that lot of places put time estimates for writing a Windows file system on the order of years. So this is pretty discouraging.

But otherwise I think it would be a pretty interesting and extremely useful project. (Ironically, it seems the harder it would be to implement FUSE the more useful it would be…) So before I give up on the idea, I thought I’d ask here if there’s any way that it’s possible. It wouldn’t have to be production quality or anything, though it would have to work somewhat reasonably well. So on a scale of 1 (“this is totally possible”) to (“you’d be more likely to create peace in the Middle East”), how crazy of an idea is this?

This would be about a two month project with two or three people. I haven’t done any file system work in the past, but I have done a non-trivial amount of Linux programming. (So I’m used to API targets that move from point-release to point-release and aren’t documented pretty much at all besides the code.) We also have the FUSE source we could use at least as a high-level guide of what they do (and I would think I might be able to get permission to actually incorporate the source into our project if it would help), so it’s not like we’d be working completely blind.

I’m also not an expert in Windows architecture or driver development, so if there’s a way to do this you think I might not see that would be nice to know as well. There are really only two restrictions other than the time we have to put in, which are that (1) I don’t want this to be just a shell namespace extension and (2) I would *really* like there to be source-compatibility with existing FUSE filesystems, possibly with the aid of Services for Unix. (There doesn’t I think need to be compatibility with the kernelspace-userspace API of FUSE; in other words, we could modify the library (that we definitely can do without license issues), perhaps with somewhat of a compatibility layer to smooth out differences such as filename matching. There just needs to be a way to build such a translation layer.)

Thanks,
Evan Driscoll

Appendix: Summary of FUSE

FUSE stands for Filesystems in UserSpaceE. It was originally written for Linux, and has since been ported to FreeBSD and OS X. FUSE has two parts, a kernel module and a userspace library, and filesystems built upon it are essentially daemon processes (services if you prefer) that link to the FUSE library to form the userspace part of a FUSE filesystem. (I’ll call the US part a server in keeping with microkernel terms.)

When a filesystem is mounted with fusermount, it starts the server. The server reads /dev/fuse, and blocks. When an application submits a I/O request to this filesystem, the request first goes to the FUSE kernel module. The kernel module does some multiplexing, and responds to the read of /dev/fuse with the request information, waking up the appropriate server. The server can then go and satisfy the request in whatever way it needs to, at which point it writes back to /dev/fuse, waking the kernel thread responding to the original application request which can then go and return the result to the application.

(At least I think this is how it works.)

There are a large number of filesystems built on FUSE, of which two interesting ones are wayback, a versioning filesystem, and sshfs, which lets you mount remote directories while it behind the scenes issues appropriate sftp commands.

Evan,

I suspect this is not a 2-3 month project, although it might be possible
to provide a proof-of-concept demonstration in that timeframe. You
would also be well-served to look at other existing user-mode file
system implementations in the Windows environment:

  • Microsoft Research at one point had a framework conceptually similar
    to what you are describing. I tried to find it on their web site but
    did not find anything obvious (which is not to suggest a more thorough
    search would not turn it up.)

  • the CMU CODA project’s Windows port is an example of a user/kernel
    file system implementation model that is based on our file system
    development toolkit.

  • the WebDAV implementation in Windows (XP and more recent) is a split
    user/kernel file system that relies upon the mini-redirector model to
    implement its functionality.

It is certainly possible there are others as well, although I may not be
aware of them, or wouldn’t consider them as this type of model (e.g.,
the OpenAFS model based on the work that Mike Kazar did 12 years ago to
graft the AFS cache manager onto an SMB server implementation.)

Best of luck on your project.

Regards,

Tony

Tony Mason
Consulting Partner
OSR Open Systems Resources, Inc.
http://www.osr.com

> When a filesystem is mounted with fusermount, it starts the server. The
server

reads /dev/fuse, and blocks. When an application submits a I/O request to this
filesystem, the request first goes to the FUSE kernel module. The kernel
module

Once I implemented something similar for Windows - alone, in 2000 or so.

It took at least 9 months to the not-so-well-debugged version, with FTP being
the stuff over which I implemented such a filesystem. Client-side caching of
downloaded files was also included.

This is just for your estimation. A large project, I would personally not take
such a a degree work with timeframes.


Maxim Shatskih, Windows DDK MVP
StorageCraft Corporation
xxxxx@storagecraft.com
http://www.storagecraft.com

We implement a model like this for SftpDrive [www.sftpdrive.com] and
it was not an easy task. We approached the project with a fairly
similar mentality and instead of 2-3 months it took closer to a year
and is still not perfectly debugged [but we never crash]. Windows
Filesystem dev is very unforgiving and I don’t recommend taking it
lightly. Easily the hardest project I’ve ever taken on, and I’d be
wary to do anything other than a rudimentary proof of concept while
under any sort of deadline

-Jeff

On 3/1/07, Maxim S. Shatskih wrote:
> > When a filesystem is mounted with fusermount, it starts the server. The
> server
> >reads /dev/fuse, and blocks. When an application submits a I/O request to this
> >filesystem, the request first goes to the FUSE kernel module. The kernel
> module
>
> Once I implemented something similar for Windows - alone, in 2000 or so.
>
> It took at least 9 months to the not-so-well-debugged version, with FTP being
> the stuff over which I implemented such a filesystem. Client-side caching of
> downloaded files was also included.
>
> This is just for your estimation. A large project, I would personally not take
> such a a degree work with timeframes.
>
> –
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@gmail.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

We implement a model like this for SftpDrive [www.sftpdrive.com] and
it was not an easy task. We approached the project with a fairly
similar mentality and instead of 2-3 months it took closer to a year
and is still not perfectly debugged [but we never crash]. Windows
Filesystem dev is very unforgiving and I don’t recommend taking it
lightly. Easily the hardest project I’ve ever taken on.

-Jeff

On 3/1/07, Maxim S. Shatskih wrote:
> > When a filesystem is mounted with fusermount, it starts the server. The
> server
> >reads /dev/fuse, and blocks. When an application submits a I/O request to this
> >filesystem, the request first goes to the FUSE kernel module. The kernel
> module
>
> Once I implemented something similar for Windows - alone, in 2000 or so.
>
> It took at least 9 months to the not-so-well-debugged version, with FTP being
> the stuff over which I implemented such a filesystem. Client-side caching of
> downloaded files was also included.
>
> This is just for your estimation. A large project, I would personally not take
> such a a degree work with timeframes.
>
> –
> Maxim Shatskih, Windows DDK MVP
> StorageCraft Corporation
> xxxxx@storagecraft.com
> http://www.storagecraft.com
>
>
> —
> Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: xxxxx@gmail.com
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

What happened to storagecrafts umfs?
I can’t seem to find it on the storagecraft site.
Wouldn’t that be ideal for something like this?

Marc Juul Christoffersen
PeerSing Organization

It’s probably a stupid question, but is it possible to use FileSystem Minifilter for such FUSE task ?
something like - get a request (IRP_MJ_CREATE/READ/WRITE/…) , forward it to user-space (FltSendMessage…) , complete the request without passing it down to the lower-FSD .

Of course, it’ll have few limitations:

  • the minifilter itself will have to create the file-object
  • no caching , no FAST IO , no PAGING IO , or any other “advanced” stuf

tnx

On Fri 02 Mar 1:40 2007 Jeff Mancuso wrote:

We implement a model like this for SftpDrive [www.sftpdrive.com] and
it was not an easy task. We approached the project with a fairly
similar mentality and instead of 2-3 months it took closer to a year
and is still not perfectly debugged [but we never crash]. Windows
Filesystem dev is very unforgiving and I don’t recommend taking it
lightly. Easily the hardest project I’ve ever taken on.

-Jeff

On 3/1/07, Maxim S. Shatskih wrote:
> > > When a filesystem is mounted with fusermount, it starts the server. The
> > server
> > >reads /dev/fuse, and blocks. When an application submits a I/O request to this
> > >filesystem, the request first goes to the FUSE kernel module. The kernel
> > module
> >
> > Once I implemented something similar for Windows - alone, in 2000 or so.
> >
> > It took at least 9 months to the not-so-well-debugged version, with FTP being
> > the stuff over which I implemented such a filesystem. Client-side caching of
> > downloaded files was also included.
> >
> > This is just for your estimation. A large project, I would personally not take
> > such a a degree work with timeframes.
> >
> > –
> > Maxim Shatskih, Windows DDK MVP
> > StorageCraft Corporation
> > xxxxx@storagecraft.com
> > http://www.storagecraft.com
> >
> >
> > —
> > Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
> >
> > You are currently subscribed to ntfsd as: xxxxx@gmail.com
> > To unsubscribe send a blank email to xxxxx@lists.osr.com
> >
>
> —
> Questions? First check the IFS FAQ at https://www.osronline.com/article.cfm?id=17
>
> You are currently subscribed to ntfsd as: ferents@cs.bgu.ac.il
> To unsubscribe send a blank email to xxxxx@lists.osr.com
>

> It’s probably a stupid question, but is it possible to use FileSystem Minifilter for such FUSE task ?

Yes and no. Yes, its possible (I’ve done it before). No, it’s not feasible. Overhead of implementing DiskDevice / Mount is probably 1% of all work, involved in implementing of the entire system. If you go “minifilter” way (actually, it must be “legasy” way, not minifilter) you will still have to do all that 99% of work plus you have all sorts of your underlying FS / filter interoperability issues.
*
Probably the easiest (fastest) way to go in this case is using miniredirector model. But still, it’s at least 6-9 month for prototyping and at least tree times as much till you get something shipable.