Device Object deletion left with references hold by NTFS threads on 2k8 R2 SP1

Hi folks,

I am facing strage problem on win 2k8 platform.My driver is a legacy driver that is loaded/unloaded on demand from Service Control Manager.My driver creates/deletes virtual volume device objects through few IOCTLs sent to control device object.
This works well on 2k3 platform.SCM unloads the driver as well.
But there are single/multiple references hold by NTFS threads on 2k8 R2 SP1. There are no explicit ref/deref in my driver but all other indirect ref/deref pairs are perfectly matched.

OS version :
Windows 7 Kernel Version 7601 (Service Pack 1) MP (2 procs) Free x64
Built by: 7601.17514.amd64fre.win7sp1_rtm.101119-1850

Test #1
!obtrace shows following stack unmatched with dereferences

53b4e1b +1 Dflt nt! ?? ::FNODOBFM::`string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16

This reference is never dereferenced by the time IoDeleteDevice() is triggered and

!object shows

1: kd> !object 0xfffffa80`034d2af0
Object: fffffa80034d2af0 Type: (fffffa8000cfa9f0) Device
ObjectHeader: fffffa80034d2ac0 (new version)
HandleCount: 0 PointerCount: 1
Directory Object: 00000000 Name: VVolume0{60746e96-a6f0-11e0-9898-000c294bcf95}

Test #2 : Another test shows that there are 6 references left

478ac76 +1 Dflt nt! ?? ::FNODOBFM::string'+1a0c7 nt!TmpIsClusteredTransactionManager+37 nt!TmpCreateLogFile+148 nt!TmpCreateOrOpenLogTransactionManager+1e nt!TmInitializeTransactionManager+1f7 nt!NtCreateTransactionManager+ea nt!KiSystemServiceCopyEnd+13 nt!KiServiceLinkage+0 Ntfs!TxfCreateTmInstance+9c Ntfs!TxfStartRm+a6a Ntfs!TxfInitializeVolume+5be Ntfs!NtfsCommonFileSystemControl+6b Ntfs!NtfsFspDispatch+2ad nt!ExpWorkerThread+111 nt!PspSystemThreadStartup+5a nt!KxStartSystemThread+16 478ac89 +1 Dflt nt! ?? ::FNODOBFM::string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs! ?? ::FNODOBFM::string'+28a3 Ntfs!TxfStartRm+a6a Ntfs!TxfInitializeVolume+5be Ntfs!NtfsCommonFileSystemControl+6b Ntfs!NtfsFspDispatch+2ad nt!ExpWorkerThread+111 nt!PspSystemThreadStartup+5a nt!KxStartSystemThread+16 478ad26 +1 Dflt nt! ?? ::FNODOBFM::string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16
478ad7d +1 Dflt nt! ?? ::FNODOBFM::`string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16

478adb0 +1 Dflt nt! ?? ::FNODOBFM::`string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16

Setting breakpoint on the “PointerCount” field does not hit breakpoint by any other component.
Because of this, IO manager does not call registered unload routine. Also SCM stop command fails, driver stucks in “STOP_PENDING” state and does not come out until reboot.

I have tried all possible options.Similar issue is reported on vista but no evidence provided who is holding.
http://www.osronline.com/showThread.cfm?link=141961

Please suggest.

Thanks,
Suresh

>My driver creates/deletes virtual volume device objects through few IOCTLs

sent to control device object.

How do you delete the volumes? Do you dismount the file system and then
delete the device, or do you surprise remove them, or something else?
Assuming that you delete these devices via some sort of normal means I’d
expect NTFS to teardown gracefully.

-scott


Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntfsd…

Hi folks,

I am facing strage problem on win 2k8 platform.My driver is a legacy driver
that is loaded/unloaded on demand from Service Control Manager.My driver
creates/deletes virtual volume device objects through few IOCTLs sent to
control device object.
This works well on 2k3 platform.SCM unloads the driver as well.
But there are single/multiple references hold by NTFS threads on 2k8 R2 SP1.
There are no explicit ref/deref in my driver but all other indirect
ref/deref pairs are perfectly matched.

OS version :
Windows 7 Kernel Version 7601 (Service Pack 1) MP (2 procs) Free x64
Built by: 7601.17514.amd64fre.win7sp1_rtm.101119-1850

Test #1
!obtrace shows following stack unmatched with dereferences

53b4e1b +1 Dflt nt! ?? ::FNODOBFM::`string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16

This reference is never dereferenced by the time IoDeleteDevice() is
triggered and

!object shows

1: kd> !object 0xfffffa80`034d2af0
Object: fffffa80034d2af0 Type: (fffffa8000cfa9f0) Device
ObjectHeader: fffffa80034d2ac0 (new version)
HandleCount: 0 PointerCount: 1
Directory Object: 00000000 Name:
VVolume0{60746e96-a6f0-11e0-9898-000c294bcf95}

Test #2 : Another test shows that there are 6 references left

478ac76 +1 Dflt nt! ?? ::FNODOBFM::string'+1a0c7 nt!TmpIsClusteredTransactionManager+37 nt!TmpCreateLogFile+148 nt!TmpCreateOrOpenLogTransactionManager+1e nt!TmInitializeTransactionManager+1f7 nt!NtCreateTransactionManager+ea nt!KiSystemServiceCopyEnd+13 nt!KiServiceLinkage+0 Ntfs!TxfCreateTmInstance+9c Ntfs!TxfStartRm+a6a Ntfs!TxfInitializeVolume+5be Ntfs!NtfsCommonFileSystemControl+6b Ntfs!NtfsFspDispatch+2ad nt!ExpWorkerThread+111 nt!PspSystemThreadStartup+5a nt!KxStartSystemThread+16 478ac89 +1 Dflt nt! ?? ::FNODOBFM::string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs! ?? ::FNODOBFM::string'+28a3 Ntfs!TxfStartRm+a6a Ntfs!TxfInitializeVolume+5be Ntfs!NtfsCommonFileSystemControl+6b Ntfs!NtfsFspDispatch+2ad nt!ExpWorkerThread+111 nt!PspSystemThreadStartup+5a nt!KxStartSystemThread+16 478ad26 +1 Dflt nt! ?? ::FNODOBFM::string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16
478ad7d +1 Dflt nt! ?? ::FNODOBFM::`string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16

478adb0 +1 Dflt nt! ?? ::FNODOBFM::`string’+1a0c7
nt!TmpIsClusteredTransactionManager+37
nt!TmpCreateLogFile+148
nt!TmpCreateOrOpenLogTransactionManager+1e
nt!TmInitializeTransactionManager+1f7
nt!NtCreateTransactionManager+ea
nt!KiSystemServiceCopyEnd+13
nt!KiServiceLinkage+0
Ntfs!TxfCreateTmInstance+9c
Ntfs!TxfStartRm+a6a
Ntfs!TxfInitializeVolume+5be
Ntfs!NtfsCommonFileSystemControl+6b
Ntfs!NtfsFspDispatch+2ad
nt!ExpWorkerThread+111
nt!PspSystemThreadStartup+5a
nt!KxStartSystemThread+16

Setting breakpoint on the “PointerCount” field does not hit breakpoint by
any other component.
Because of this, IO manager does not call registered unload routine. Also
SCM stop command fails, driver stucks in “STOP_PENDING” state and does not
come out until reboot.

I have tried all possible options.Similar issue is reported on vista but no
evidence provided who is holding.
http://www.osronline.com/showThread.cfm?link=141961

Please suggest.

Thanks,
Suresh

I’ve run into the same issue. This is a windows bug. You’ll need to follow
up with MS. I’ll send you more details offline.

Thanks,
Alex.

Ouch, good to know.

-scott


Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

“Alex Carp” wrote in message news:xxxxx@ntfsd…

I’ve run into the same issue. This is a windows bug. You’ll need to follow
up with MS. I’ll send you more details offline.

Thanks,
Alex.

>> Assuming that you delete these devices via some sort of normal means
I’d expect NTFS to teardown gracefully.
Scott : yes, this is true.

Thanks Alex, for quick response. I have seen this issue on Win 2008 SP2
also. Sure, I would be in touch with MS on this and post updates on this
forum.

Thanks,
Suresh
On Thu, Jul 14, 2011 at 7:13 PM, Alex Carp wrote:

> I’ve run into the same issue. This is a windows bug. You’ll need to follow
> up with MS. I’ll send you more details offline.
>
> Thanks,
> Alex.
>
>
> —
> NTFSD is sponsored by OSR
>
> For our schedule of debugging and file system seminars visit:
> http://www.osr.com/seminars
>
> To unsubscribe, visit the List Server section of OSR Online at
> http://www.osronline.com/page.cfm?name=ListServer
>