FiFo in KMDF 1.9

NTDEV
1

Hello
Is there a simple way for a FiFo Buffer for inter-thread-comunication (only in the Driver, not to the user mode!!)
Can i use NamedPipe in Kernal Driver (with ZwRead,…)

Or is there a Wdf-function for a FiFo Buffer similar NamedPipe?

kBrause

2

No. Yes. Maybe.

Instead of asking us about a specific mechanism, can you please describe the overall problem you’re trying to solve? That’ll help us give you the best answer.

Please start with “I have two threads, THREAD A and THREAD B. I need …”

Peter
OSR

3

OK:
i’m lazy and NamedPipe is a 100% working FiFo between diff. threads (with lock for sync!)
when i’m not lazy, i would imlements my own RingBuffer…

maybe: is a manuel serial WDFQueue an option??

Requirements:
Store Binary MSG variable lenght.
Persisten (context DriverObjekt)
serial (FiFo)
inter-thread-comunication

4

Why not use a double linked list? Very simple in a driver

Put a LIST_ENTRY list head in your context. Put a KSPIN_LOCK there for locking
When you allocate a buffer for your MSG, make sure there is a LIST_ENTRY in your structure.
When inserting, grab the lock, call InsertHeadList(&listhead, &msg->ListEntry), drop the lock
When removing, grab the lock, PLIST_ENTRY ple = RemoveHeadList(&listhead); PSMG msg = CONTAINING_RECORD(ple, MSG, ListEntry), drop the lock

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@t-online.de
Sent: Saturday, March 17, 2012 10:54 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] FiFo in KMDF 1.9

OK:
i’m lazy and NamedPipe is a 100% working FiFo between diff. threads (with lock for sync!) when i’m not lazy, i would imlements my own RingBuffer…

maybe: is a manuel serial WDFQueue an option??

Requirements:
Store Binary MSG variable lenght.
Persisten (context DriverObjekt)
serial (FiFo)
inter-thread-comunication

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

5

why not isn’t the question:
only the question:
whats the best, safe (and laziest) way for my requirements??

OK, i’ll have a look at “InsertHeadList”

and thanks!

6

The linked list is stupid simple, you don’t have to read twice for a var sized packet (header for size, buffer) like you would with a pipe. I would use the linked list 99.99% of the time.

d

debt from my phone

From: xxxxx@t-online.de
Sent: 3/17/2012 1:18 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] FiFo in KMDF 1.9

why not isn’t the question:
only the question:
whats the best, safe (and laziest) way for my requirements??

OK, i’ll have a look at “InsertHeadList”

and thanks!

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

7

thanks in advanced!! super easy, super lazy

8

Only for the records:
FiFi: Insert at the bottom, take form the top (InsertTailList, RemoveHeadList)

9

FiFo not FiFi :lol:

10

awhen using a linked list, one of the questions is often “how do I notify
my thread that there is sonething in the queue?” All too often, the answer
is KEVENT, but the correct answer is KSEMAPHORE.
joe

The linked list is stupid simple, you don’t have to read twice for a var
sized packet (header for size, buffer) like you would with a pipe. I would
use the linked list 99.99% of the time.

d

debt from my phone

From: xxxxx@t-online.de
Sent: 3/17/2012 1:18 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] FiFo in KMDF 1.9

why not isn’t the question:
only the question:
whats the best, safe (and laziest) way for my requirements??

OK, i’ll have a look at “InsertHeadList”

and thanks!

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

11

I’ve seen this advice several times now, and on the surface of it, I
don’t see why a semaphore is required, and an event won’t work. I’m
sure when you explain it, I’ll slap my forehead and exclaim “DUH!”, so
I’ll get the ball rolling by asking the question: “Why?”

Cheers,

–mkj

On 3/25/2012 5:28 PM, xxxxx@flounder.com wrote:

awhen using a linked list, one of the questions is often “how do I notify
my thread that there is sonething in the queue?” All too often, the answer
is KEVENT, but the correct answer is KSEMAPHORE.
joe

> The linked list is stupid simple, you don’t have to read twice for a var
> sized packet (header for size, buffer) like you would with a pipe. I would
> use the linked list 99.99% of the time.
>
> d
>
> debt from my phone
> ________________________________
> From: xxxxx@t-online.de
> Sent: 3/17/2012 1:18 PM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] FiFo in KMDF 1.9
>
> why not isn’t the question:
> only the question:
> whats the best, safe (and laziest) way for my requirements??
>
>
> OK, i’ll have a look at “InsertHeadList”
>
> and thanks!
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

–

//
// Michael K. Jones
// Stone Hill Consulting, LLC
// http://www.stonehill.com
//_______________________________________________

12

Since semaphores are counting items, for every insert you increment the
semaphore and every item you take out you decrement the semaphore. If
you try to do this with an event you need a some additional
synchronization to ensure that the event is set as long as there is a
item in the fifo. Of course it is a little obscure and the interface is
a little weird, but Windows does provide a FIFO with signaling built in
see KeInitializeQueue, KeInsertQueue and KeRemoveQueue.

Don Burn
Windows Filesystem and Driver Consulting
Website: http://www.windrvr.com
Blog: http://msmvps.com/blogs/WinDrvr

“Michael Jones” wrote in message
news:xxxxx@ntdev:

> I’ve seen this advice several times now, and on the surface of it, I
> don’t see why a semaphore is required, and an event won’t work. I’m
> sure when you explain it, I’ll slap my forehead and exclaim “DUH!”, so
> I’ll get the ball rolling by asking the question: “Why?”
>
> Cheers,
>
> --mkj
>
> On 3/25/2012 5:28 PM, xxxxx@flounder.com wrote:
> > awhen using a linked list, one of the questions is often “how do I notify
> > my thread that there is sonething in the queue?” All too often, the answer
> > is KEVENT, but the correct answer is KSEMAPHORE.
> > joe
> >
> >> The linked list is stupid simple, you don’t have to read twice for a var
> >> sized packet (header for size, buffer) like you would with a pipe. I would
> >> use the linked list 99.99% of the time.
> >>
> >> d
> >>
> >> debt from my phone
> >>
> >> From: xxxxx@t-online.de
> >> Sent: 3/17/2012 1:18 PM
> >> To: Windows System Software Devs Interest List
> >> Subject: RE:[ntdev] FiFo in KMDF 1.9
> >>
> >> why not isn’t the question:
> >> only the question:
> >> whats the best, safe (and laziest) way for my requirements??
> >>
> >>
> >> OK, i’ll have a look at “InsertHeadList”
> >>
> >> and thanks!
> >>
> >>
> >>
> >> —
> >> NTDEV is sponsored by OSR
> >>
> >> For our schedule of WDF, WDM, debugging and other seminars visit:
> >> http://www.osr.com/seminars
> >>
> >> To unsubscribe, visit the List Server section of OSR Online at
> >> http://www.osronline.com/page.cfm?name=ListServer
> >>
> >>
> >> —
> >> NTDEV is sponsored by OSR
> >>
> >> For our schedule of WDF, WDM, debugging and other seminars visit:
> >> http://www.osr.com/seminars
> >>
> >> To unsubscribe, visit the List Server section of OSR Online at
> >> http://www.osronline.com/page.cfm?name=ListServer
> >
> >
> >
>
> –
>_______________
> //
> // Michael K. Jones
> // Stone Hill Consulting, LLC
> // http://www.stonehill.com
> // _______________________________________________

13

C’mon… It’ll work either way. You need to lock around the list insertion and removal ANYway, so if you use the right event type, and take the lock and clear the event in the right place, an event will work just fine.

Peter
OSR

14

I always use an event…

d

debt from my phone

From: Michael Jones
Sent: 3/26/2012 6:10 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] FiFo in KMDF 1.9

I’ve seen this advice several times now, and on the surface of it, I
don’t see why a semaphore is required, and an event won’t work. I’m
sure when you explain it, I’ll slap my forehead and exclaim “DUH!”, so
I’ll get the ball rolling by asking the question: “Why?”

Cheers,

–mkj

On 3/25/2012 5:28 PM, xxxxx@flounder.com wrote:

awhen using a linked list, one of the questions is often “how do I notify
my thread that there is sonething in the queue?” All too often, the answer
is KEVENT, but the correct answer is KSEMAPHORE.
joe

> The linked list is stupid simple, you don’t have to read twice for a var
> sized packet (header for size, buffer) like you would with a pipe. I would
> use the linked list 99.99% of the time.
>
> d
>
> debt from my phone
> ________________________________
> From: xxxxx@t-online.de
> Sent: 3/17/2012 1:18 PM
> To: Windows System Software Devs Interest List
> Subject: RE:[ntdev] FiFo in KMDF 1.9
>
> why not isn’t the question:
> only the question:
> whats the best, safe (and laziest) way for my requirements??
>
>
> OK, i’ll have a look at “InsertHeadList”
>
> and thanks!
>
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>
>
> —
> NTDEV is sponsored by OSR
>
> For our schedule of WDF, WDM, debugging and other seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer

–

//
// Michael K. Jones
// Stone Hill Consulting, LLC
// http://www.stonehill.com
//_______________________________________________

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

15

Thank you, Doron and Peter, for preventing a bruised forehead. I
certainly get why a semaphore would work; I just thought I was missing
something re: using an event. Glad to know there’s no mystery (just
careful coding required, as always).

Cheers,

–mkj

On 3/26/2012 10:11 AM, Doron Holan wrote:

I always use an event…

d

debt from my phone

From: Michael Jones
Sent: 3/26/2012 6:10 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] FiFo in KMDF 1.9

I’ve seen this advice several times now, and on the surface of it, I
don’t see why a semaphore is required, and an event won’t work. I’m
sure when you explain it, I’ll slap my forehead and exclaim “DUH!”, so
I’ll get the ball rolling by asking the question: “Why?”

Cheers,

–mkj

On 3/25/2012 5:28 PM, xxxxx@flounder.com wrote:
> awhen using a linked list, one of the questions is often “how do I notify
> my thread that there is sonething in the queue?” All too often, the answer
> is KEVENT, but the correct answer is KSEMAPHORE.
> joe
>
> > The linked list is stupid simple, you don’t have to read twice for a var
> > sized packet (header for size, buffer) like you would with a pipe. I
would
> > use the linked list 99.99% of the time.
> >
> > d
> >
> > debt from my phone
> > ________________________________
> > From: xxxxx@t-online.de
> > Sent: 3/17/2012 1:18 PM
> > To: Windows System Software Devs Interest List
> > Subject: RE:[ntdev] FiFo in KMDF 1.9
> >
> > why not isn’t the question:
> > only the question:
> > whats the best, safe (and laziest) way for my requirements??
> >
> >
> > OK, i’ll have a look at “InsertHeadList”
> >
> > and thanks!
> >
> >
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
> >
> >
> > —
> > NTDEV is sponsored by OSR
> >
> > For our schedule of WDF, WDM, debugging and other seminars visit:
> > http://www.osr.com/seminars
> >
> > To unsubscribe, visit the List Server section of OSR Online at
> > http://www.osronline.com/page.cfm?name=ListServer
>
>
>

–

//
// Michael K. Jones
// Stone Hill Consulting, LLC
// http://www.stonehill.com
//_______________________________________________

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

–

//
// Michael K. Jones
// Stone Hill Consulting, LLC
// http://www.stonehill.com
//_______________________________________________

16

> C’mon… It’ll work either way. You need to lock around the list insertion and removal ANYway

You can use ExInterlocked list + a semaphore, which is probably faster then spinlock+double linked list+KEVENT.

–
Maxim S. Shatskih
Windows DDK MVP
xxxxx@storagecraft.com
http://www.storagecraft.com

17

Peter,

I know you can do it, and I know most synchronization experts can do it,
but when somebody asks such a basic question, the important thing is to
make sure they get it right the first time. The number of failures I’ve
seen in user space, where the programmer tried to use an event instead of
a mutex or semaphore, is amazingly high. And in one of the MFC
newsgroups, this question came up about once a month, accompanied by code
that rarely took more than one reading to find the failure condition.

I’ve seen one in the kernel where a spin lock was set, and held by the
consumer thread until the queue was empty. And at the insertion site, the
programmer did an InterlockedIncrement of the count and THEN did the
InterlockedInsertTailList. BSOD was the result. Mostly, though, they
simply end up in a race condition about setting and resetting the event.
Except for the guy who set a spin lock right before the KeWaitFor…, and
couldn’t understand the problem. “I’m in a driver thread”, he said, “So
I’m at PASSIVE_LEVEL”. So I long ago learned to give the simplest
solution that works “out of the box”. Watching someone debug event code
into correctness is also pretty frightening; I spent three days at one
site where hach after hack was added to the event-based simulation of a
semaphore, because the programmer said “semaphores are too complicated to
understand”. On Friday, his manager came in, demanded to know what
progress had been made, got seriosly bent out of shape at no progress,
turned to me (the outside consultant) and said “Can you help?” I ripped
out all his synchronization code, and replaced it with two lines: one to
put something in an IOCP and one to remove sonething from the IOCP. I had
it working in about ten minutes. Then, it turned out, they had to run it
on MS-DOS laptops (aka Win95), and the manager was pissed at my solution.
So I rewrote it in 15 minutes, using CRITICAL_SECTION and semaphore, and
CList in MFC. So I learned long ago that most people cannot reason about
concurrency, and the secret of success is to eliminate the need for them
to do so.
joe

C’mon… It’ll work either way. You need to lock around the list
insertion and removal ANYway, so if you use the right event type, and take
the lock and clear the event in the right place, an event will work just
fine.

Peter
OSR

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer

18

On 26-Mar-2012 20:39, xxxxx@flounder.com wrote:
…

I learned long ago that most people cannot reason about
concurrency, and the secret of success is to eliminate the need for them
to do so.

So true!
( Hello, serialized NDIS and similar stuff in the WDF )

– pa

19

First question: why do you see a semaphore as “unnatural” and an event as
“natural”?
Second question: why do you feel compelled to re-implement a semaphore
from low-level components like events?
Third question: how is a homebrewed solution superior to an existing,
documented, supported mechanism?

Most people seem clueless as to the role of semaphores, but I’ve been
writing semaphore-based code in 1968 and don’t see anything surprising
here. The notion of having to simulate a semaphore with counters and
events is about at the same level as the people who say “I don’t need no
stinkin’ spin lock, why, I can build my own just with this here LOCK
prefix” and don’t get what is really going on. And occasionally, they
actually manage to get it right, which proves only that they love wasting
time reinventing wheels.

If I have a problem for which a semaphore is the right answer, I don’t try
to simulate it; I just use the capability that is already there.
Interthread queues are a prime example. In what way is an event superior
to a semaphore?

Note that at application level, I don’t even use semaphores for
interthread queueing; I/O Completion Ports do it all, with zero effort on
my part. I only used semaphores in MS-DOS (aka Win9x) because there were
no IOCPs.
joe

I’ve seen this advice several times now, and on the surface of it, I
don’t see why a semaphore is required, and an event won’t work. I’m
sure when you explain it, I’ll slap my forehead and exclaim “DUH!”, so
I’ll get the ball rolling by asking the question: “Why?”

Cheers,

–mkj

On 3/25/2012 5:28 PM, xxxxx@flounder.com wrote:
> awhen using a linked list, one of the questions is often “how do I
> notify
> my thread that there is sonething in the queue?” All too often, the
> answer
> is KEVENT, but the correct answer is KSEMAPHORE.
> joe
>
>> The linked list is stupid simple, you don’t have to read twice for a
>> var
>> sized packet (header for size, buffer) like you would with a pipe. I
>> would
>> use the linked list 99.99% of the time.
>>
>> d
>>
>> debt from my phone
>> ________________________________
>> From: xxxxx@t-online.de
>> Sent: 3/17/2012 1:18 PM
>> To: Windows System Software Devs Interest List
>> Subject: RE:[ntdev] FiFo in KMDF 1.9
>>
>> why not isn’t the question:
>> only the question:
>> whats the best, safe (and laziest) way for my requirements??
>>
>>
>> OK, i’ll have a look at “InsertHeadList”
>>
>> and thanks!
>>
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> For our schedule of WDF, WDM, debugging and other seminars visit:
>> http://www.osr.com/seminars
>>
>> To unsubscribe, visit the List Server section of OSR Online at
>> http://www.osronline.com/page.cfm?name=ListServer
>
>
>

–

//
// Michael K. Jones
// Stone Hill Consulting, LLC
// http://www.stonehill.com
//_______________________________________________

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at
http://www.osronline.com/page.cfm?name=ListServer