SL_PENDING_RETURNED

Hello,

My understanding through some experiment is PendingReturned once set does not get reset to zero. In that case all the upper driver will set its control to SL_PENDING_RETURNED. This can happen in case any driver in the stack makes the IO to synchronous

What’s the use case of SL_PENDING_RETURNED for IoManager? As PendingReturned is already in place.

e.g. IRP passes through following stack
Driver A->Driver B->Driver C->Driver D->Driver E

Driver D
Dispatch:-MarkIrpPending
return Pending

Driver C
Dispatch:-
IoCopyCurrentIrpStacklocation
IoSetCompletionRoutine
Status = IoCallDriver
if (Status == Status_Pending) {
Wait
Status = Irp.Iostatus.Status;
}
IoCompleteRequest
return Status

Driver C Completion Routine:-
Set Event
Status More…P…R

Driver B
Dispatch
IoCopyCurrentIrpStacklocation
IoSetCompletionRoutine
return IoCallDriver
Completion
if (Irp->PendingReturned)
IoMArkPEnding
return Status_success;

Whether APC is queued by IoCompletionRoutine post the last completion routine in the stack?
Whether it makes its decision on PendingReturned or SL_PENDING_RETURNED?

I have gone through
http://www.osronline.com/showThread.cfm?link=89035
http://www.osronline.com/article.cfm?article=83

“My understanding through some experiment is PendingReturned once set does not get reset to zero.”
this is mistake. really in IofCompleteRequestin loop (by IRP stacks) exist line:
Irp->PendingReturned = stackPointer->Control & SL_PENDING_RETURNED;
so PendingReturned every time recalculated based on stackPointer->Control & SL_PENDING_RETURNED

IoMarkIrpPending marks the *current* stack position with SL_PENDING_RETURNED, for the upper level caller.

PendingReturned is set from SL_PENDING_RETURNED of the *next* stack position (set by the lower driver), for the completion routine, just before the *next* stack position is zeroed out and the completion routine is called.

A completion routine cannot examine SL_PENDING_RETURNED, it examines PendingReturned.

A system top level completion routine which ultimately calls ExQueueAPC can only examine PendingReturned.

“A completion routine cannot examine SL_PENDING_RETURNED, it examines
PendingReturned.” - i say about system routine in ntoskrnl - IofCompleteRequest

say in your example - driver D set SL_PENDING_RETURNED.
IofCompleteRequestin set Irp->PendingReturned to TRUE and call Driver C Completion Routine
Driver C Completion Routine - return STATUS_MORE_PROCESSING_REQUIRED - as result IofCompleteRequestin exit at this point
then Driver C dispatch routine take control after awaken from Waiting and call IoCompleteRequest again
but in current stack now - SL_PENDING_RETURNED not set !
IofCompleteRequestin set Irp->PendingReturned to FALSE and call Driver B Completion Routine

You can only return STATUS_MORE_PROCESSING_REQUIRED from a completion routine if you are serializing back to your dispatch handler (KeWaitForSingleObject on an object you set in the completion routine) OR you marked the IRP as pending prior to returning it. Those have been the rules since NT 3.1 and they are the rules today.

This is because the logic around completion is bifurcated - in one place the processing only has the status code to look at and in the other place it only has the IRP to look at. That’s why we have these rules: to ensure the two code paths in the I/O Manager make the same decision. If you give back one answer via the status and a different via the IRP, it breaks.

Tony
OSR

This is all explained very nicely in an article Tony wrote 20 years ago for The NT Insider on this topic. Guess what? Aside from the name of the OS, nothing else has really changed:

http:

I apologize for the hideous formatting of the article online. It’s one of the most useful articles ever written. We should do some editing and move it to the OSR.COM at some point.

Peter
OSR
@OSRDrivers</http:>

Thanks for input.

In my understanding only the top layer will have impact.
Whats a impact in case intermediate driver ignore or set it explicitly? As in dispatch it is returning status pending explicitly.

Case:- Irp is marked pending and returned status_pending. Later the irp is pass down to lower stack through private thread. As in this case ideally completion routine should check for PendingReturned but ignores assuming upper layer driver would be setting the flag appropriately.

I’m guessing you didn’t bother reading the article I pointed you to, right?

Well, in this case the completion routine is broken, right? And the system is broken as a result.

To restate what’s already been said, perhaps in more clear terms: EACH driver handling an IRP in a completion routine, in that completion routine must check Irp->PendingReturned… and if it’s set, call IoMarkIrpPending.

If a driver DOES NOT specify a completion routine, the I/O Manager provides one that does the required step I described above.

Irp->PendingReturned only reflects whether the NEXT LOWER driver called IoMarkIrpPending or not.

Improperly handling I/O Completion in Windows *was* one of the most common causes of crashes, prior to the introduction of WDF. WDF has made this problem “go away” for most developers… but if you’re using WDM, it’s one of the most complex topics to understand.

Peter
OSR
@OSRDrivers

>EACH driver handling an IRP in a completion routine, in that completion routine must check
Irp->PendingReturned… and if it’s set, call IoMarkIrpPending.

EXCEPT for the completion routine set by the top level driver which created the IRP, and a completion routine which returns STATUS_MORE_PROCESSING_REQUIRED. The top level completion routine does NOT have a current stack location, and an attempt to call IoMarkIrpPending will cause memory corruption.

The rule here is that if a dispatch routine passes the IoCallDriver result to the return from the dispatch routine, it MUST do the “if (PendingReturned) IoMarkIrpPending(Irp)” combination in its completion routine.

STATUS_PENDING return from dispatch routine MUST match SL_PENDING_RETURNED in its stack location.

Yes, yes… IF a top level driver created the IRP. There’d be no point in a driver creating an IRP calling IoMarkIrpPending, because he’s not returning status to anyone.

… OR ignores the returned result and returns STATUS_PENDING…

Peter
OSR
@OSRDrivers

You can’t call IoMarkIrpPending on a driver created irp in the creating driver, there is no current valid stack location in the irp so you would be trashing memory

Sent from Outlook Mailhttp: for Windows 10 phone

From: xxxxx@osr.com
Sent: Tuesday, October 6, 2015 7:44 PM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] SL_PENDING_RETURNED

Yes, yes… IF a top level driver created the IRP. There’d be no point in a driver creating an IRP calling IoMarkIrpPending, because he’s not returning status to anyone.



… OR ignores the returned result and returns STATUS_PENDING…

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</http:>

> My understanding through some experiment is PendingReturned once set does not get reset to

zero.

Yes.

More so, the magic snippet of:

if( Irp->PendingReturned )
IoMarkIrpPending(Irp);

which is mandatory for all completion routines returning STATUS_SUCCESS, propagates the once set PendingReturned up to the “threaded IRP completion path” in the IO manager.

In that case all the upper driver will set its control to SL_PENDING_RETURNED. This can happen in
case any driver in the stack makes the IO to synchronous

Yes. Note that IoMarkIrpPending just sets SL_PENDING_RETURNED, and nothing else.

What’s the use case of SL_PENDING_RETURNED for IoManager?

SL_PENDING_RETURNED/Irp->PendingReturned is usually always set, except the synchronous IRPs (where some dispatch routine completed the IRP inline, which is usually the case if there was a wait involved).

This influences IoCompleteRequest.

IoCompleteRequest (callable in any process/thread context) has a second step called IopCompleteRequest, which is called on APC_LEVEL in the exact process/thread context which submitted the IRP.

If the IRP is synchronous, i.e. SL_PENDING_RETURNED/Irp->PendingReturned is NOT set, then, according to the well-known rule, status other than STATUS_PENDING must be returned from the dispatch routines.

In this case, the IopSynchronousServiceTail function (“submit the threaded IRP and process its completion”) just calls IopCompleteRequest inline after IoCallDriver, without queuing it as the APC.

Surely, for this, the APC must not be queued from IoCompleteRequest either (to avoid duplicate call to IopCompleteRequest). And, for this, IoCompleteRequest checks the SL_PENDING_RETURNED/Irp->PendingReturned stuff in the IRP.

If the IRP is NOT synchronous, then the dispatch path returns STATUS_PENDING, which tells IopSynchronousServiceTail to NOT call IopCompleteRequest directly.

Also, in this case, SL_PENDING_RETURNED/Irp->PendingReturned is set, so IoCompleteRequest will queue IopCompleteRequest as an APC.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> IofCompleteRequestin set Irp->PendingReturned to FALSE

I don’t think IoCompleteRequest ever sets Irp->PendingReturned to FALSE.

It becomes TRUE forever if it was set at least once.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> In my understanding only the top layer will have impact.

Whats a impact in case intermediate driver ignore or set it explicitly?

This whole stuff comes into play only when the threaded IRP is completed by all drivers back up to the IO manager (which originally created it).

The intermediate drivers are only required to:
a) call IoMarkIrpPending in all paths where STATUS_PENDING is returned, more so, do this before putting the IRP to the context (queue or such) from where it can be completed asynchronously.
AND
b) propagate the flag in the completion routines.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

>prior to the introduction of WDF. WDF has made this problem “go away” for most developers…

Oh yes, WDF just returns STATUS_PENDING for everything.

The WDM driver can also do the same, like:

IoMarkIrpPending
(VOID)IoCallDriver
return STATUS_PENDING;


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

> EXCEPT for the completion routine set by the top level driver which created the IRP,

Why “except”? if the driver has created the IRP, then it is not threaded one, it never goes to IopCompleteRequest and is just IoFreeIrp’ed in the CR.

and a completion routine which returns STATUS_MORE_PROCESSING_REQUIRED.

Imagine:

  • Upper calls Lower for the first time
  • upper returns the status of Lower, which is STATUS_PENDING
  • the Upper’s CR (CR1) called by Lower resubmits the IRP back to Lower and returns STATUS_MORE_PROCESSING_REQUIRED
  • the 2nd call to Lower does the processing inline and does not set the pending flag

In your case, CR1 will not propagate the pending flag. The possible CR2 will not propagate it either since it is not set when CR2 sees the IRP.

So, STATUS_PENDING is returned to IO with pending flag NOT set -> a crash.


Maxim S. Shatskih
Microsoft MVP on File System And Storage
xxxxx@storagecraft.com
http://www.storagecraft.com

>> IofCompleteRequestin set Irp->PendingReturned to FALSE

I don’t think IoCompleteRequest ever sets Irp->PendingReturned to FALSE.
It becomes TRUE forever if it was set at least once.
you mistake - look for IofCompleteRequestin src code of for disassembler

was LOOP by stack locations and in LOOP was line:
Irp->PendingReturned = stackPointer->Control & SL_PENDING_RETURNED;

so PendingReturned calculated several times, for every stack location. so it can be changed.
if say some driver returned STATUS_MORE_PROCESSING_REQUIRED from self Completion(after KeSetEvent for which wait in dispatch routine) and it dispatch routine return 0 or some error status, but not mark self stack with SL_PENDING_RETURNED

No. Well, not in the code I’m looking at in any case. It sets Irp->PendingReturned before calling the completion routine based on whether the NEXT LOWER driver’s I/O stack Location has SL_PENDING_RETURNED set.

It wouldn’t make sense otherwise.

I/O Manger-> Driver A -> Driver B.

If DriverA sets a completion routine and IoCallDriver’s an IRP to DriverB, DriverA is under no requirement whatsoever to return STATUS_PENDING. It might BLOCK after the IoCallDriver, and then complete the Request with STATUS_SUCCESS. If Irp->PendingReturned was set, this would call IoCompleteRequest to queue the Special Kernel APC for I/O Completion… This would result in I/O Completion Stage 2 (IopCompleteRequest) running twice. A bad thing.

[The problem is that this is a complex topic, and we really can’t coherently discuss it in short posts, structured as replies to a specific question. I would suggest that we’re not likely to get anything useful with this thread, other than getting a lot of people confused when they read the archives in the future.]

Given the New Open Microsoft, it would be VERY useful if short sections or selected pieces of the I/O Manger could be made publicly available. It would put to rest a lot of myth about this topic, for example.

Peter
OSR
@OSRDrivers

“This would result in I/O Completion Stage 2 (IopCompleteRequest) running twice.”
really IopCompleteRequest (Stage #2) off course must be called once only. but it in general must be called in originate Process context (if request from user mode) unlike IopfCompleteRequest (Stage #1) - which can be called in any process context.
IopCompleteRequest (Stage #2) is called direct, without APC, if (Irp->Flags & IRP_DEFER_IO_COMPLETION) && !Irp->PendingReturned
otherwise will be used APC for call IopCompleteRequest (Stage #2) in right process context.
of course direct call more effective than APC usage.