Wait-wake IRP completion and IoCancelIrp race

NTDEV
1

Hi,

I have 2 drivers (bus and function) and a piece of the HW in the following stack:

HW

BUS

FUNCTION

The BUS driver is receives the wake message from the HW and simultaneously the system enters the S0.

From this point the flow is as follows:

  1. BUS driver transits from D3 ->D0.
  2. FUNCTION driver starts D3->D0 entry and registers completion routine.
  3. BUS driver completes the Wait - wake IRP (WdfDeviceIndicateWakeStatus) as a result of the HW event.
  4. Completion routine from pt 2 in FUNCTION driver is called and the driver calls IoCancelIrp on the wait wake IRP. The IRQL is LOW_LEVEL and the call is on processor 0.
  5. Completion callback resulting from pt 3. - wait wake completion is called. The IRQL is DISPATCH and the call is on the processor 2.
  6. The wait wake completion callback ends.
  7. The IoCancelIrp fails with the BSOD 0xA.

Upon analysis it turns out that the IRP is freed, the memory is paged out (verifier) as IoCancelIrp is trying to set the Cancel flag in the IRP.

Is there a method to synchronize the above flow?

I’ve found the same issue (from 2004), when reviewing the OSR online, but it was unanswered (http://www.osronline.com/showThread.cfm?link=113692).

Best Regards,
Marek Dabek

2

http://blogs.msdn.com/b/doronh/archive/2008/06/29/how-do-i-cancel-an-irp-that-another-thread-may-be-completing-at-the-same-time.aspx

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@intel.com
Sent: Tuesday, March 31, 2015 8:28 AM
To: Windows System Software Devs Interest List
Subject: [ntdev] Wait-wake IRP completion and IoCancelIrp race

Hi,

I have 2 drivers (bus and function) and a piece of the HW in the following stack:

HW

BUS

FUNCTION

The BUS driver is receives the wake message from the HW and simultaneously the system enters the S0.

From this point the flow is as follows:

  1. BUS driver transits from D3 ->D0.
  2. FUNCTION driver starts D3->D0 entry and registers completion routine.
  3. BUS driver completes the Wait - wake IRP (WdfDeviceIndicateWakeStatus) as a result of the HW event.
  4. Completion routine from pt 2 in FUNCTION driver is called and the driver calls IoCancelIrp on the wait wake IRP. The IRQL is LOW_LEVEL and the call is on processor 0.
  5. Completion callback resulting from pt 3. - wait wake completion is called. The IRQL is DISPATCH and the call is on the processor 2.
  6. The wait wake completion callback ends.
  7. The IoCancelIrp fails with the BSOD 0xA.

Upon analysis it turns out that the IRP is freed, the memory is paged out (verifier) as IoCancelIrp is trying to set the Cancel flag in the IRP.

Is there a method to synchronize the above flow?

I’ve found the same issue (from 2004), when reviewing the OSR online, but it was unanswered (http://www.osronline.com/showThread.cfm?link=113692).

Best Regards,
Marek Dabek

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

3

Doron,

Thank you for your answer.
Does your explanation applies to the power IRPs as well?

I am using the PoRequestPowerIrp to request the IRP. The MSDN states:
“The PowerCompletion routine performs any additional tasks the sender of the IRP requires after all other drivers have completed the IRP. It need not free the IRP; the power manager does that.”

Since the FUNCTION driver is not freeing the IRP directly I am not able to control when IoFreeIrp should be called.

Is there a way to control power manager on when should it free the IRP?

Best regards,
Marek Dabek

4

You control when you complete the wait wake power request. You can defer completion outside of the completion routine if needed to coordinate with the cancelation attempt

d

Bent from my phone

From: xxxxx@intel.commailto:xxxxx
Sent: ?4/?1/?2015 7:31 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Wait-wake IRP completion and IoCancelIrp race

Doron,

Thank you for your answer.
Does your explanation applies to the power IRPs as well?

I am using the PoRequestPowerIrp to request the IRP. The MSDN states:
“The PowerCompletion routine performs any additional tasks the sender of the IRP requires after all other drivers have completed the IRP. It need not free the IRP; the power manager does that.”

Since the FUNCTION driver is not freeing the IRP directly I am not able to control when IoFreeIrp should be called.

Is there a way to control power manager on when should it free the IRP?

Best regards,
Marek Dabek

—
NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

5

Doron,

Thank you again.

One more question. Since I am completing the Wait Wake IRP using WdfDeviceIndicateWakeStatus, the completion routine has IRQL == DISPATCH_LEVEL, this makes waiting impossible.

Can I force the WdfDeviceIndicateWakeStatus to not complete the IRP on the DISPATCH_LEVEL?

6

No, you can’t change irql or force it to passive. The beauty of the ref count solution is that you don’t have to synch wait, the thread which decrements the count to zero knows it is the owner and can complete the request

d

Bent from my phone

From: xxxxx@intel.commailto:xxxxx
Sent: ?4/?1/?2015 1:10 PM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Wait-wake IRP completion and IoCancelIrp race

Doron,

Thank you again.

One more question. Since I am completing the Wait Wake IRP using WdfDeviceIndicateWakeStatus, the completion routine has IRQL == DISPATCH_LEVEL, this makes waiting impossible.

Can I force the WdfDeviceIndicateWakeStatus to not complete the IRP on the DISPATCH_LEVEL?

—
NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

7

Doron,

Basing on what you wrote, the solution I see would be:

  1. Increment completion counter right before calling IoCancelIrp
  2. If the counter before the update was non 0, drop cancelling.
  3. In completion routine (which is a result of the Wait Wake IRP complete in another driver) increment the completion counter.
  4. If the counter before the update was non 0, decrement it and execute tight loop (wait locks not possible since completion may be executed on the dispatch level).
  5. When the counter drops to 0, continue with the finishing completion callback.

However, I see 2 opens in the above solution:

  1. If IoCancelIrp and completion routine execute on the same processor I hit the deadlock.
  2. What happens if the IRP is cancelled and completion callaback continues, eventually returning. Would the power manager start working on already freed IRP?

Best regards,
Marek

8

There are no tight loops, the completion routine would decrement the count, not increment

d

Bent from my phone

From: xxxxx@intel.commailto:xxxxx
Sent: ?4/?2/?2015 7:12 AM
To: Windows System Software Devs Interest Listmailto:xxxxx
Subject: RE:[ntdev] Wait-wake IRP completion and IoCancelIrp race

Doron,

Basing on what you wrote, the solution I see would be:
1. Increment completion counter right before calling IoCancelIrp
2. If the counter before the update was non 0, drop cancelling.
3. In completion routine (which is a result of the Wait Wake IRP complete in another driver) increment the completion counter.
4. If the counter before the update was non 0, decrement it and execute tight loop (wait locks not possible since completion may be executed on the dispatch level).
5. When the counter drops to 0, continue with the finishing completion callback.

However, I see 2 opens in the above solution:
1. If IoCancelIrp and completion routine execute on the same processor I hit the deadlock.
2. What happens if the IRP is cancelled and completion callaback continues, eventually returning. Would the power manager start working on already freed IRP?

Best regards,
Marek

—
NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer</mailto:xxxxx></mailto:xxxxx>

9

Doron,

I reviewed the thread and there are 2 things I am missing here.

  1. In the original solution you proposed the counter was ensuring that the IoFreeIrp will be called once. This is not the case here since the wait wake IRP is power IRP. According to the MSDN :
    “The PowerCompletion routine should not free the IRP?the power manager does that.”.
    Why should counter protect in my case?

  2. The wait wake irp cancelling and completing take part in two different driver. What is the driver which completes the irp is using WDF to do so (WdfDeviceIndicateWakeStatus), which means the driver is not using the IRP in the explicitly.
    I assume that the way to synchronize cancelling and completing the wait wake irp between drivers this is quite challenging (implementation-wise) and an over engineering.
    Is there a better way to ensure the race-free cancel / complete flow of the power irps?

Best regards,
Marek

10

1 you are guarding against the io completion set via IoSetCompletionRoutine (not the power request completion routine supplied to PoRequestPowerIrp) from completing the irp (returning anything other than STATUS_MORE_PROCESSING_REQUIRED) when calling IoCancelIrp.

2 not exactly. Remember that each driver in the stack that sets a completion routine on the irp completes the request. While bus driver is the first to complete it (WdfDeviceIndicateWakeStatus), any driver above it also completes the WW irp as it travels up the stack and completion routines are called. If you are calling IoCancelIrp, you need to have a completion routine registered and use the reference count to make sure that the race between canceling the request and your completion routine completing the request do not race

Is there a better way to ensure the race-free cancel / complete flow of the power irps?

WDM irp cancelation is hard, I gave you the easiest, simplest path. You need to understand how IRPs flow through a device stack more thoroughly.

d

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@intel.com
Sent: Tuesday, April 07, 2015 2:24 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Wait-wake IRP completion and IoCancelIrp race

Doron,

I reviewed the thread and there are 2 things I am missing here.

  1. In the original solution you proposed the counter was ensuring that the IoFreeIrp will be called once. This is not the case here since the wait wake IRP is power IRP. According to the MSDN :
    “The PowerCompletion routine should not free the IRP?the power manager does that.”.
    Why should counter protect in my case?

  2. The wait wake irp cancelling and completing take part in two different driver. What is the driver which completes the irp is using WDF to do so (WdfDeviceIndicateWakeStatus), which means the driver is not using the IRP in the explicitly.
    I assume that the way to synchronize cancelling and completing the wait wake irp between drivers this is quite challenging (implementation-wise) and an over engineering.
    Is there a better way to ensure the race-free cancel / complete flow of the power irps?

Best regards,
Marek

NTDEV is sponsored by OSR

Visit the list at: http://www.osronline.com/showlists.cfm?list=ntdev

OSR is HIRING!! See http://www.osr.com/careers

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer