Upper filter not loading on Win10 RS2 (version 1703)

Hello folks,

I have a PnP upper filter for usb removable disk devices, which does the usual stuff:
creates device object, attaches to target device during AddDevice via IoAttachDeviceToDeviceStack, detaches during IRP_MN_REMOVE_DEVICE, and so on.

This has been working fine all the way from XP to 8.1, but now i am seeing an odd behavior, and only on Windows 10 RS2 (version 1703).
This is what I’m seeing:

  • filter loads and works fine the first time I plug the thumbdrive.
  • when i unplug the usb drive, driver detaches and unloads ok.
  • when i plug again the thumbdrive, filter won’t load and device manager tells me there is a previous instance (error code 38)

I can see with Windbg that the driver is not really unloaded, and with !obtrace I can see there are some outstanding references, but I can’t see much more information on those stack traces.

Maybe MS changed the way it manages the device attachment stuff for Windows 10 Creators Update? Are you aware of any changes related to this area?

Thanks in advance!

I tried to decrement “manually” the reference cound for the device object used for attaching, by putting an additional ObDereferenceObject call, and now the filter is unloading and loading without problems, so apparently IoAttachDeviceToDeviceStack has a different behavior on Win10 RS2 ?

There is no indication of that behaviour (increased reference count) in the documentation except when the target device object pointer is obtained with IoGetDeviceObjectPointer.

You should make a check with a pass-through upper filter based on the Toaster Filter sample for example.

H. G.

1703 has some filter driver issues. I am tracking down an issue right now.
I’ll post more when I know more. I get a BSOD, and it makes no sense.

On Fri, Jun 30, 2017 at 10:39 AM wrote:

> There is no indication of that behaviour (increased reference count) in
> the documentation except when the target device object pointer is obtained
> with IoGetDeviceObjectPointer.
>
>
>
> You should make a check with a pass-through upper filter based on the
> Toaster Filter sample for example.
>
>
>
> H. G.
>
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

>1703 has some filter driver issues.

That is a very disturbing possibility… I HOPE this turns out to not be the case.

Peter
OSR
@OSRDrivers

I started looking into this last night. Here is what is odd: The driver
installs as an upper volume filter. The registry shows:

volsnap
OurDriver

By accident, testers discovered if a space is placed in front of the
OurDriver, everything works fine:

volsnap
OurDriver

Right?

The issue appears in build 1703. Last night I tried to install Windows 10
x64 1703 in VirtualBox, and the system freezes soon after the OS starts to
load from the ISO image. VBOX version 5.0.40_Ubuntu r115130. I tried BIOS
and EFI. I’ll need to setup another physical test environment to track this
one down.

On Fri, Jun 30, 2017 at 1:53 PM wrote:

> >1703 has some filter driver issues.
>
> That is a very disturbing possibility… I HOPE this turns out to not be
> the case.
>
> Peter
> OSR
> @OSRDrivers
>
>
> —
> NTDEV is sponsored by OSR
>
> Visit the list online at: <
> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>
> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
> software drivers!
> Details at http:
>
> To unsubscribe, visit the List Server section of OSR Online at <
> http://www.osronline.com/page.cfm?name=ListServer&gt;
></http:>

To understand this correctly, you are saying if the class upper filters is “volsnap”,“OurDriver” the system doesn’t boot or your driver doesn’t load. But if the multi sz is “volsnap”," OurDriver" the system works as expected?

d

Bent from my phone


From: xxxxx@lists.osr.com on behalf of Jamey Kirby
Sent: Friday, June 30, 2017 11:15:14 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Upper filter not loading on Win10 RS2 (version 1703)

I started looking into this last night. Here is what is odd: The driver installs as an upper volume filter. The registry shows:

volsnap
OurDriver

By accident, testers discovered if a space is placed in front of the OurDriver, everything works fine:

volsnap
OurDriver

Right?

The issue appears in build 1703. Last night I tried to install Windows 10 x64 1703 in VirtualBox, and the system freezes soon after the OS starts to load from the ISO image. VBOX version 5.0.40_Ubuntu r115130. I tried BIOS and EFI. I’ll need to setup another physical test environment to track this one down.

On Fri, Jun 30, 2017 at 1:53 PM > wrote:
>1703 has some filter driver issues.

That is a very disturbing possibility… I HOPE this turns out to not be the case.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:>

To unsubscribe, visit the List Server section of OSR Online at http:>
— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:>

OurDriver loads in both cases In the first case, it gets a BSOD at some
later time while the system is running (during VSS creation). But, I get no
BSOD in the second case. The dump file indicates a spinlock might have been
held too long, and the call stack shows it is when OurDriver is calling
PsCreateSystemThread. I need to duplicate in my environment before I can
dig deeper.

On Fri, Jun 30, 2017 at 2:15 PM Jamey Kirby wrote:

> I started looking into this last night. Here is what is odd: The driver
> installs as an upper volume filter. The registry shows:
>
> volsnap
> OurDriver
>
> By accident, testers discovered if a space is placed in front of the
> OurDriver, everything works fine:
>
> volsnap
> OurDriver
>
> Right?
>
> The issue appears in build 1703. Last night I tried to install Windows 10
> x64 1703 in VirtualBox, and the system freezes soon after the OS starts to
> load from the ISO image. VBOX version 5.0.40_Ubuntu r115130. I tried BIOS
> and EFI. I’ll need to setup another physical test environment to track this
> one down.
>
>
>
> On Fri, Jun 30, 2017 at 1:53 PM wrote:
>
>> >1703 has some filter driver issues.
>>
>> That is a very disturbing possibility… I HOPE this turns out to not be
>> the case.
>>
>> Peter
>> OSR
>> @OSRDrivers
>>
>>
>> —
>> NTDEV is sponsored by OSR
>>
>> Visit the list online at: <
>> http://www.osronline.com/showlists.cfm?list=ntdev&gt;
>>
>> MONTHLY seminars on crash dump analysis, WDF, Windows internals and
>> software drivers!
>> Details at http:
>>
>> To unsubscribe, visit the List Server section of OSR Online at <
>> http://www.osronline.com/page.cfm?name=ListServer&gt;
>>
></http:>

Sounds more like a race condition than a configuration problem if the driver is loading with and without the space.

d

From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Jamey Kirby
Sent: Friday, June 30, 2017 11:52 AM
To: Windows System Software Devs Interest List
Subject: Re: [ntdev] Upper filter not loading on Win10 RS2 (version 1703)

OurDriver loads in both cases In the first case, it gets a BSOD at some later time while the system is running (during VSS creation). But, I get no BSOD in the second case. The dump file indicates a spinlock might have been held too long, and the call stack shows it is when OurDriver is calling PsCreateSystemThread. I need to duplicate in my environment before I can dig deeper.

On Fri, Jun 30, 2017 at 2:15 PM Jamey Kirby > wrote:
I started looking into this last night. Here is what is odd: The driver installs as an upper volume filter. The registry shows:

volsnap
OurDriver

By accident, testers discovered if a space is placed in front of the OurDriver, everything works fine:

volsnap
OurDriver

Right?

The issue appears in build 1703. Last night I tried to install Windows 10 x64 1703 in VirtualBox, and the system freezes soon after the OS starts to load from the ISO image. VBOX version 5.0.40_Ubuntu r115130. I tried BIOS and EFI. I’ll need to setup another physical test environment to track this one down.

On Fri, Jun 30, 2017 at 1:53 PM > wrote:
>1703 has some filter driver issues.

That is a very disturbing possibility… I HOPE this turns out to not be the case.

Peter
OSR
@OSRDrivers


NTDEV is sponsored by OSR

Visit the list online at: http:>

MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers!
Details at http:>

To unsubscribe, visit the List Server section of OSR Online at http:>
— NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars on crash dump analysis, WDF, Windows internals and software drivers! Details at To unsubscribe, visit the List Server section of OSR Online at</http:></http:></http:>

This is a usual behavior when you have some sort of “incompatibility” with another driver that holds a reference to a device object or a driver object. Usually there is a bug in third party driver that reveals itself with a new filer. I don’t have any recollection of such problem with any MS driver but non MS drivers suffer from it quite often.

> The dump file indicates a spinlock might have been held too long, and the
call stack shows it is when OurDriver is calling PsCreateSystemThread.

PsCreateSystemThread.is a PASSIVE_LEVEL routine. Don’t you have a code
analysis warning ? You can’t call this routine while a spin-lock is held.

Not yet. I have to setup a test/debug environment for this situation. I am
reporting on initial dump file analysis and reports from test department.
I’ll know more after the weekend.

On Sat, Jul 1, 2017 at 2:47 PM hgfhhgf jhgfgbbgf
wrote:

> > The dump file indicates a spinlock might have been held too long, and
> the call stack shows it is when OurDriver is calling PsCreateSystemThread.
>
> PsCreateSystemThread.is a PASSIVE_LEVEL routine. Don’t you have a code
> analysis warning ? You can’t call this routine while a spin-lock is held.
> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
> on crash dump analysis, WDF, Windows internals and software drivers!
> Details at To unsubscribe, visit the List Server section of OSR Online at

I’m not calling it with a spinlock help. That is what is odd.

On Sat, Jul 1, 2017 at 7:37 PM Jamey Kirby wrote:

> Not yet. I have to setup a test/debug environment for this situation. I am
> reporting on initial dump file analysis and reports from test department.
> I’ll know more after the weekend.
>
> On Sat, Jul 1, 2017 at 2:47 PM hgfhhgf jhgfgbbgf
> wrote:
>
>> > The dump file indicates a spinlock might have been held too long, and
>> the call stack shows it is when OurDriver is calling PsCreateSystemThread.
>>
>> PsCreateSystemThread.is a PASSIVE_LEVEL routine. Don’t you have a code
>> analysis warning ? You can’t call this routine while a spin-lock is held.
>> — NTDEV is sponsored by OSR Visit the list online at: MONTHLY seminars
>> on crash dump analysis, WDF, Windows internals and software drivers!
>> Details at To unsubscribe, visit the List Server section of OSR Online
>> at
>
>

Hi again,

So I have been investigating this issue, making all sort of changes, removing unnecessary operations from my filter, until having the minimum stuff for attaching and detaching, but the problem is still happening.

My driver is based on the WDM Toaster class upper filter included in WDK 7, so I removed my filter and put that sample (CLSUPPER.SYS) instead, and the issue is also reproduced with that driver.

Note that, just in case, I also tried modifying the WDK sample so, during IRP_MN_REMOVE_DEVICE, the call to IoReleaseRemoveLockAndWait would be made AFTER passing the irp down with IoCallDriver, but the problem still occurs.

So are you aware of any bug in the WDM version for the Toaster upper filter sample? Any wrong behavior arising only in Win10 Rs2?

I know the recommended way of filtering would be KMDF but at this moment I can’t afford porting my driver…any thoughts?

Thanks in advance!

FWIW, the problem goes away if I use Verifier on the filter…sounds like a race condition, but where?

Thanks!

Since Toaster WDM filter is also leaking a reference, I have tried this: put the additional call to ObDereferenceObject, during IRP_MN_REMOVE_DEVICE, after detachment and device deletion, and then right after that I break with debugger and then put a memory breakpoint, on the object reference count pointer (it is 0x18 bytes before DEVICE_OBJECT ptr) like this:

ba w4 0x82d52e88 “k;g”

After that, I can see these 2 stacks:

ChildEBP RetAddr

00 8b7d7968 8132447f nt!ObfDereferenceObjectWithTag+0x25
01 8b7d79a4 814b5423 nt!IopSynchronousCall+0xcb
02 8b7d7a00 811ea11f nt!IopRemoveDevice+0xe3
03 8b7d7a2c 81499c56 nt!PnpRemoveLockedDeviceNode+0x1f5
04 8b7d7a40 81499d6d nt!PnpDeleteLockedDeviceNode+0x63
05 8b7d7a80 814997ea nt!PnpDeleteLockedDeviceNodes+0xb9
06 8b7d7ab0 81499bdd nt!PipRemoveDevicesInRelationList+0x61
07 8b7d7acc 81499b49 nt!PnpDelayedRemoveWorker+0x6c
08 8b7d7ae8 811ed1f7 nt!PnpChainDereferenceComplete+0xe4
09 8b7d7b14 814ad8e9 nt!PnpIsChainDereferenced+0xcc
0a 8b7d7b88 8142abc3 nt!PnpProcessQueryRemoveAndEject+0x80c
0b 8b7d7ba8 8138be9a nt!PnpProcessTargetDeviceEvent+0xf6c31
0c 8b7d7be8 810de216 nt!PnpDeviceEventWorker+0x1f8
0d 8b7d7c38 8110bbec nt!ExpWorkerThread+0xc0
0e 8b7d7c70 81186cf5 nt!PspSystemThreadStartup+0x4a
0f 8b7d7c7c 00000000 nt!KiThreadStartup+0x15

ChildEBP RetAddr

00 8b7d79f8 810c4ece nt!ObfDereferenceObjectWithTag+0x25
01 8b7d7a00 811ea1b3 nt!ObfDereferenceObject+0x12
02 8b7d7a2c 81499c56 nt!PnpRemoveLockedDeviceNode+0x289
03 8b7d7a40 81499d6d nt!PnpDeleteLockedDeviceNode+0x63
04 8b7d7a80 814997ea nt!PnpDeleteLockedDeviceNodes+0xb9
05 8b7d7ab0 81499bdd nt!PipRemoveDevicesInRelationList+0x61
06 8b7d7acc 81499b49 nt!PnpDelayedRemoveWorker+0x6c
07 8b7d7ae8 811ed1f7 nt!PnpChainDereferenceComplete+0xe4
08 8b7d7b14 814ad8e9 nt!PnpIsChainDereferenced+0xcc
09 8b7d7b88 8142abc3 nt!PnpProcessQueryRemoveAndEject+0x80c
0a 8b7d7ba8 8138be9a nt!PnpProcessTargetDeviceEvent+0xf6c31
0b 8b7d7be8 810de216 nt!PnpDeviceEventWorker+0x1f8
0c 8b7d7c38 8110bbec nt!ExpWorkerThread+0xc0
0d 8b7d7c70 81186cf5 nt!PspSystemThreadStartup+0x4a
0e 8b7d7c7c 00000000 nt!KiThreadStartup+0x15

But I’m not sure I understand the meaning of this, because it’s the system who is touching the count…any ideas?

Can you unplug and then replug the usb stick when your filter isn’t loaded ?

Can you unplug and then replug the usb stick with a pass-through filter ?

Have you read this ?

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/object-reference-tracing

Yes, I can unplug and then replug the usb stick with my filter unloaded, no problem.

With the Toaster passthru filter, I can unplug the stick but when I replug it again then the driver can’t load, because of the leaked reference. That’s the whole point of this issue.

Remember, it also happens with Toaster WDM filter, not only with my driver.

And yep, I have used the !obtrace extension, but the output didn’t throw much light on the subject.

Thanks for your help!

This is weird. The KMDF version of Toaster filter is also showing the same behavior.

It attaches the first time I plug the thumbdrive, but if I unplug and replug again, then the filter doesn’t load because of outstanding references.

Any thoughts?

Javier,

How does the whole stack looks like? Do you install it on ‘clean’ OS? Is that possible that some user mode app opens a handle that pins the DO and the DRVO in memory? Also, you could try to put a hardware bp on access to DO’s ReferenceCount and then kb on every stop. I guess you won’t get too many stops, so it looks doable and might shed some light on who references you.