User / Kernel concurrent IOCTLs

Hello,

Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?

If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.
This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.
I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.

Thanks.
Shay

Take a look at IoCompletionPorts
http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@emc.com” wrote in message
news:xxxxx@ntdev:

> Hello,
>
> Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?
>
> If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.
> This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.
> I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.
>
> Thanks.
> Shay

>I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their >completion.
You could use one thread per IOCTL in an application.

Igor Sharovar

Do you know if Linux has the same notion of IO completion ports ?

There is a big chance this code will later need to be converted to run on Linux so I was trying to ease the pain for later with some solution which has a Linux equivalent for it.

Shay

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, July 11, 2011 4:05 PM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] User / Kernel concurrent IOCTLs

Take a look at IoCompletionPorts
http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@emc.com” wrote in message
news:xxxxx@ntdev:

> Hello,
>
> Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?
>
> If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.
> This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.
> I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.
>
> Thanks.
> Shay


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

No Linux does not. Encapsulate the handling of the I/O and use
completion ports. I have worked on several projects where the code was
to be used on Linux and Windows and the attempts to share a code base in
a fast I/O path have always produced lousy performance.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@emc.com” wrote in message
news:xxxxx@ntdev:

> Do you know if Linux has the same notion of IO completion ports ?
>
> There is a big chance this code will later need to be converted to run on Linux so I was trying to ease the pain for later with some solution which has a Linux equivalent for it.
>
> Shay
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
> Sent: Monday, July 11, 2011 4:05 PM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] User / Kernel concurrent IOCTLs
>
> Take a look at IoCompletionPorts
> http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
>
> “xxxxx@emc.com” wrote in message
> news:xxxxx@ntdev:
>
> > Hello,
> >
> > Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?
> >
> > If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.
> > This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.
> > I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.
> >
> > Thanks.
> > Shay
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Also look at the thread pool APIs which abstract io completion ports a little bit, plus it allows for other waits. It has a little bit of a learning curve, but so does completion ports.

d

debt from my phone

-----Original Message-----
From: Don Burn
Sent: Monday, July 11, 2011 1:33 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Re:User / Kernel concurrent IOCTLs

No Linux does not. Encapsulate the handling of the I/O and use
completion ports. I have worked on several projects where the code was
to be used on Linux and Windows and the attempts to share a code base in
a fast I/O path have always produced lousy performance.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@emc.com” wrote in message
news:xxxxx@ntdev:

> Do you know if Linux has the same notion of IO completion ports ?
>
> There is a big chance this code will later need to be converted to run on Linux so I was trying to ease the pain for later with some solution which has a Linux equivalent for it.
>
> Shay
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
> Sent: Monday, July 11, 2011 4:05 PM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] User / Kernel concurrent IOCTLs
>
> Take a look at IoCompletionPorts
> http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
>
> “xxxxx@emc.com” wrote in message
> news:xxxxx@ntdev:
>
> > Hello,
> >
> > Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?
> >
> > If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.
> > This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.
> > I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.
> >
> > Thanks.
> > Shay
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

Thanks for the quick reply. I’ll take a look at the IO ports to see if it gives me better encapsulation option than the solution I have today.

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
Sent: Monday, July 11, 2011 4:30 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Re:User / Kernel concurrent IOCTLs

No Linux does not. Encapsulate the handling of the I/O and use
completion ports. I have worked on several projects where the code was
to be used on Linux and Windows and the attempts to share a code base in
a fast I/O path have always produced lousy performance.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

xxxxx@emc.com” wrote in message
news:xxxxx@ntdev:

> Do you know if Linux has the same notion of IO completion ports ?
>
> There is a big chance this code will later need to be converted to run on Linux so I was trying to ease the pain for later with some solution which has a Linux equivalent for it.
>
> Shay
>
>
> -----Original Message-----
> From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Don Burn
> Sent: Monday, July 11, 2011 4:05 PM
> To: Windows System Software Devs Interest List
> Subject: Re:[ntdev] User / Kernel concurrent IOCTLs
>
> Take a look at IoCompletionPorts
> http://msdn.microsoft.com/en-us/library/aa365198(VS.85).aspx
>
>
> Don Burn
> Windows Filesystem and Driver Consulting
> Website: http://www.windrvr.com
> Blog: http://msmvps.com/blogs/WinDrvr
>
>
>
>
> “xxxxx@emc.com” wrote in message
> news:xxxxx@ntdev:
>
> > Hello,
> >
> > Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?
> >
> > If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.
> > This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.
> > I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.
> >
> > Thanks.
> > Shay
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

>This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I

have to somehow remember all the handles I got and wait for them to complete.

Use IO completion ports in this case. Google for “GetQueuedCompletionStatus”.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

>Do you know if Linux has the same notion of IO completion ports ?

No.

Since “aio” support in UNIXen was historically pathetic compared to Windows, UNIXen go the other, also elegant, way, i.e. select() and poll().

Instead of sending many IOCTLs, pending them in the driver, completing them in the driver when something occurs, and then using IOCP to wait on them - your UNIX driver listens for these “something occurs” events and remembers them in its queue.

Then the user app calls poll() on the driver, i.e. wait for the driver’s queue to become non-empty.

Then the user app (note: it can be single threaded) submits an IOCTL, which is guaranteed to be executed in a nonblocking way in such a case.

poll() can also be used to wait on lots of drivers simultaneously.

This is the classic difference between Windows and UNIXen (the classic case is accepting incoming TCP connections actually).

So, invent the abstract “event listener” object, which will be different on UNIXen and Windows, and write the rest of user mode code around it.


Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

On 11/07/2011 22:27, Maxim S. Shatskih wrote:

> Do you know if Linux has the same notion of IO completion ports ?
No.

Since “aio” support in UNIXen was historically pathetic compared to Windows, UNIXen go the other, also elegant, way, i.e. select() and poll().

The high-performance aio solution in Linux is libaio
(http://lse.sourceforge.net/io/aio.html):

“AIO enables even a single application thread to overlap I/O operations
with other processing, by providing an interface for submitting one or
more I/O requests in one system call (io_submit()) without waiting for
completion, and a separate interface (io_getevents()) to reap completed
I/O operations associated with a given completion group.”


Bruce Cran

For new code, I suggest the thread pool APIs. They are build on top of IO completion ports and they abstract much of the plumbing and system specific coding (ex how many worker threads should I spawn & what concurrency should I use). Using these APIs means that you can use code that Microsoft has already built and debugged, so unless you feel like doing exrta work it only makes sense to use them. Also, it is very easy to build a shim for *NIX that uses poll / select and has the same semantic interface (and since I last did this there may be libraries available)

As an aside, overlapped IO is the way to send multiple asynchronous IOCTLs. How you choose to monitor the completion of these requests is part of the app design; there are many choices, but IMHO IOCP is the design that makes the most sense and is best scalable on SMP. NUMA machines present additional challenges, but that is a topic for another day.

wrote in message news:xxxxx@ntdev…
Hello,

Is there a way to send asynchronous IOCTLs from user to kernel w/o using overlapped IOs ?

If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use WaitForSingleObject with the handle to the event I got when I sent the IOCTL.

This complicates things if I want to send multiple IOCTLs before waiting the completion of any of them. It means I have to somehow remember all the handles I got and wait for them to complete.

I am well aware that due to user/kernel limitation we can’t have a simple completion function but I was wondering if there is a more elegant way to send concurrent IOCTLs and wait for their completion.

Thanks.

Shay

> The high-performance aio solution in Linux is libaio (http://lse.sourceforge.net/io/aio.html):

This library relies upon native Linux AIO-related system calls. Sounds great…

The “only” problem is that almost no one in the kernel tree actually implements aio_read() and aio_write() methods that these systems calls rely upon in truly asynch way. Linux drivers/subsystems/FS normally either simply delegate asynch reads and writes to the synch versions straight away, or rely upon dedicated kernel threads that process IO synchronously behind the scenes, i.e. do things exactly the same way UM AIO implementation in GLIBC does. For the fun of doing it, try to search kernel sources for kick_iocb() and aio_complete() (any truly asynchronous implementation of the aio_read() and aio_write() methods would make these calls somewhere in the driver’s code), so that you will have a chance to see with your own eyes how “popular” these calls are among driver writers…

Anton Bassov

WFSO on a handle is a practice which is actively discouraged by Microsoft,
for the reasons you state. There are two alternatives, one is to create an
Event object, which allows you to WaitForMultipleObjects (WFMO) on the array
of handles for the events, but a better solution is to use an I/O Completion
Port, where a thread handles the completion notifications and you identify
the I/O request via its OVERLAPPED object. It works like this

typedef struct {
OVERLAPPED ovl;
.your fields here.
} MYOVERLAPPED, *PMYOVERLAPPED;

PMYMOVERLAPPED myovl = new PMYOVERLAPPED;
or
PMYOVERLAPPED myovl = (PMYOVERLAPPED)malloc(sizeof(MYOVERLAPPED));

[The first is for C++ apps, the second for C apps]

Now, fill in the fields, in particular, make sure that the

myovl->ovl.hEvent = NULL;

and that the high and low values are set to 0 if you are not using them.
Then do

SomeIoOperation(,., myovl);

Now, in your handler, you do

UINT MyHandlerThread(LPVOID)
{
while(TRUE)
{ /* handler loop */
PMYOVERLAPPED myovl;
DWORDS bytesTransferred;
ULONG_PTR key;

BOOL b = GetQueuedCompletionStatus(iocp,
&bytesTransferred,
&key,
(LPOVERLAPPED)&myovl,
INFINITE);
if(!b)
{
// deal with error
}
else
{
use the key, bytesTransferred, and fields in myovl to deal
with that came back, e.g.,
if(key == INVALID_HANDLE_VALUE)
break;
.other stuff
delete myovl; // or in C, free(myovl);
} /* handler loop */
return 0;
} // MyHandlerThread

This is a sample. To shut the loop down, I do
PostQueuedCompletionStatus(iocp, 0,
(UINT_PTR)INVALID_HANDLE_VALUE, NULL);

Essentially, once you decide how you are encoding your keys (and I most
commonly use the handle as the IOCP key, so it will never be the out-of-band
INVALID_HANDLE_VALUE), you know which key values are illegal, and you
designate one of those to be your shutdown event. You can read more about
IOCPs on my MVP Tips site, www.flounder.com/mvp_tips.htm.
joe


From: xxxxx@lists.osr.com
[mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@emc.com
Sent: Monday, July 11, 2011 3:58 PM
To: Windows System Software Devs Interest List
Subject: [ntdev] User / Kernel concurrent IOCTLs

Hello,

Is there a way to send asynchronous IOCTLs from user to kernel w/o using
overlapped IOs ?

If I use FILE_FLAG_OVERLAPPED when I use CreateFile, I still have to use
WaitForSingleObject with the handle to the event I got when I sent the
IOCTL.
This complicates things if I want to send multiple IOCTLs before waiting the
completion of any of them. It means I have to somehow remember all the
handles I got and wait for them to complete.
I am well aware that due to user/kernel limitation we can’t have a simple
completion function but I was wondering if there is a more elegant way to
send concurrent IOCTLs and wait for their completion.

Thanks.
Shay


NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer