Driver Problems? Questions? Issues?
Put OSR's experience to work for you! Contact us for assistance with:
  • Creating the right design for your requirements
  • Reviewing your existing driver code
  • Analyzing driver reliability/performance issues
  • Custom training mixed with consulting and focused directly on your specific areas of interest/concern.
Check us out. OSR, the Windows driver experts.

OSR Seminars


Go Back   OSR Online Lists > ntdev
Welcome, Guest
You must login to post to this list
  Message 1 of 5  
31 Jan 18 11:44
Leo Yuriev
xxxxxx@yuriev.ru
Join Date: 31 Jan 2018
Posts To This List: 2
Seems the new kernel bug - race between NtLockFile() and NtWriteFile() could hang any multi-threaded application

Hi, Seems the new kernel bug discovered (Windows 8..10, other not checked). (unfortunately I not found other acceptable way to submit bugreport to MS). In general - a race between NtLockFile() and NtWriteFile() could hang any multi-threaded application. Testcase to reproduce https://gist.github.com/leo-yuriev/b25697898a754dbaa0dac89abf824a72 Example of kernel stack traces: Thread #1: nt!KiSwapContext+0x7a nt!KiCommitThreadWait+0x1d2 nt!KeWaitForSingleObject+0x19f nt!IopSynchronousServiceTail+0x2a9 nt!NtLockFile+0x514 nt!KiSystemServiceCopyEnd+0x13 Thread #2: nt!KiSwapContext+0x7a nt!KiCommitThreadWait+0x1d2 nt!KeWaitForSingleObject+0x19f nt!IopAcquireFileObjectLock+0x84 nt! ?? ::NNGAKEGL::`string'+0x491d5 nt!KiSystemServiceCopyEnd+0x13 Regargs, Leonid. P.S. A bug was found when testing https://github.com/leo-yuriev/libmdbx and https://github.com/leo-yuriev/libfpta --
  Message 2 of 5  
01 Feb 18 18:37
rbmm
xxxxxx@gmail.com
Join Date: 26 May 2017
Posts To This List: 3
Seems the new kernel bug - race between NtLockFile() and NtWriteFile() could hang any multi-threaded application

no, this is not window bug. this your logic bug. 2 threads call in loop: LockFileEx - WriteFile - UnlockFile now let be next execution sequence: thread_1 thread_2 ------------------------- LockFileEx LockFileEx WriteFile so 1. thread_1:LockFileEx - ok 2. thread_1:SwitchToThread(); 3. thread_2:LockFileEx all synchronous operations on file is sequential here because file object open in synchronous mode (FileObject->Flags & FO_SYNCHRONOUS_IO) IopAcquireFastLock or IopAcquireFileObjectLock for FileObject called FileObject->Busy = TRUE; IopSynchronousServiceTail IoCallDriver FsRtlProcessFileLock FsRtlPrivateCheckForExclusiveLockAccess return false: lock not Granted because it already hold by thread_1 FsRtlProcessFileLock return STATUS_PENDING because lock not Granted we return to IopSynchronousServiceTail because SynchronousIo==true and STATUS_PENDING returned IopSynchronousServiceTail call KeWaitForSingleObject( &FileObject->Event..) wait when lock irp will be completed but it will not be completed until thread_1 call UnlockFile so thread_2 begin wait when thread_1 call UnlockFile 4. thread_1:WriteFile here because file object open in synchronous mode (FileObject->Flags & FO_SYNCHRONOUS_IO) IopAcquireFastLock or IopAcquireFileObjectLock for FileObject called because FileObject->Busy == TRUE (it set by thread_2 in NtLockFile) we begin wait on until previous synchronous io request (thread_2:LockFileEx) not finished so we wait on KeWaitForSingleObject( &FileObject->Lock) what we have ? thread_1 wait until previous io operation on file will be completed but it will be complete when lock will be granted for thread_2 but this will be only after thread_1 call UnlockFile but this will be only after thread_1 finish WriteFile so thread_1 can not finish WriteFile until he not call UnlockFile but it can not call UnlockFile until it not finish WriteFile here thread_1 lock himself. and demo code for this can be much more simply and clear ULONG WINAPI thread_proc(HANDLE hFile) { OVERLAPPED ov = {}; if (LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK, 0, 1, 0, &ov)) { UnlockFile(hFile, 0, 0, 1, 0); } return GetLastError(); } HANDLE hFile = CreateFileW(L"\\\\?\\D:\\tmp.txt", FILE_GENERIC_WRITE, 0, 0, OPEN_EXISTING, 0, 0); if (hFile != INVALID_HANDLE_VALUE) { OVERLAPPED ov = {}; if (LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK, 0, 1, 0, &ov)) { if (HANDLE hThread = CreateThread(0, 0, thread_proc, hFile, 0, 0)) { // give time to thread_proc call LockFileEx MessageBoxW(0, 0, L"Now thread Hung!", MB_ICONWARNING); ULONG NumberOfBytesWritten; WriteFile(hFile, "*", 1, &NumberOfBytesWritten, 0); CloseHandle(hThread); } UnlockFile(hFile, 0, 0, 1, 0); } CloseHandle(hFile); }
  Message 3 of 5  
01 Feb 18 21:24
rbmm
xxxxxx@gmail.com
Join Date: 26 May 2017
Posts To This List: 3
Seems the new kernel bug - race between NtLockFile() and NtWriteFile() could hang any multi-threaded application

note - the hung was only because thread_1 and thread_2 use the same synchronous file handle. because synchronous io is sequential per file file object. if threads will be use separate file handles(not duplicated) - will be no hang. because separate file object locks we can change demo code in next way - not pass handle to second thread but let it open file himself. ULONG WINAPI thread_proc(HANDLE hFile) { hFile = CreateFileW(L"*", FILE_GENERIC_WRITE, FILE_SHARE_WRITE|FILE_SHARE_READ, 0, OPEN_EXISTING, 0, 0); if (hFile != INVALID_HANDLE_VALUE) { OVERLAPPED ov = {}; if (LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK, 0, 1, 0, &ov)) { MessageBoxW(0, L"inside lock", L"thread_proc", MB_ICONINFORMATION); UnlockFile(hFile, 0, 0, 1, 0); } CloseHandle(hFile); } return GetLastError(); } HANDLE hFile = CreateFileW(L"*", FILE_GENERIC_WRITE, FILE_SHARE_WRITE|FILE_SHARE_READ, 0, OPEN_EXISTING, 0, 0); if (hFile != INVALID_HANDLE_VALUE) { HANDLE hThread = 0; OVERLAPPED ov = {}; if (LockFileEx(hFile, LOCKFILE_EXCLUSIVE_LOCK, 0, 1, 0, &ov)) { if (hThread = CreateThread(0, 0, thread_proc, 0, 0, 0)) { // give time to thread_proc call LockFileEx MessageBoxW(0, 0, L"Now thread Not Hung!", MB_ICONINFORMATION); ULONG NumberOfBytesWritten; if (!WriteFile(hFile, "*", 1, &NumberOfBytesWritten, 0)) { GetLastError(); } } UnlockFile(hFile, 0, 0, 1, 0); } if (hThread) { WaitForSingleObject(hThread, INFINITE); CloseHandle(hThread); } CloseHandle(hFile); } this show that this is not a problem in case 2 separate applications, which used separate file handle. code: 1. thread_1:LockFileEx(h1) 2. thread_2:LockFileEx(h2) 3. thread_1:WriteFile(h1) 4. thread_1:UnlockFile(h1) assume that h1 -> FileObject1 != FileObject2 <- h2 will be executed without problems. thread_2 will be wait until thread_1 not call UnlockFile(h1), but this not prevent thread_1 from complete call WriteFile(h1), because FileObject1->Busy == FALSE. call LockFileEx(h2) set FileObject2->Busy = TRUE. so we not hung in IopAcquireFileObjectLock 2. not hold 3. but next code always hung (in all windows versions): 1. thread_1:LockFileEx(h1) 2. thread_2:LockFileEx(h1) 3. thread_1:WriteFile(h1) 4. thread_1:UnlockFile(h1) thread_1 will be wait inside WriteFile - IopAcquireFileObjectLock on FileObject->Lock so: 3. finished only only after 2. 2. finished only after 4. 4. finished (even begin) only after 3.
  Message 4 of 5  
01 Feb 18 23:58
Leo Yuriev
xxxxxx@yuriev.ru
Join Date: 31 Jan 2018
Posts To This List: 2
Seems the new kernel bug - race between NtLockFile() and NtWriteFile() could hang any multi-threaded application

Thank you for explanation. Not a bug, but feature ;) Seems mad for me in comparison Linux.
  Message 5 of 5  
02 Feb 18 02:35
rbmm
xxxxxx@gmail.com
Join Date: 26 May 2017
Posts To This List: 3
Seems the new kernel bug - race between NtLockFile() and NtWriteFile() could hang any multi-threaded application

but this is well known example with 2 critical section deadlock every (even query it name) synchronous operation on FileObject executed inside critical section WriteFile(FileObject) is EnterCriticalSection(&FileObject->Lock); WriteFileNoLock(FileObject); LeaveCriticalSection(&FileObject->Lock); the Lock/Unlock file exclusive - this is another critical section acquire (by sense) so when you execute LockFileEx - WriteFile - UnlockFile you execute next code: EnterCriticalSection(&FileObject->Lock); EnterCriticalSection(&cs); LeaveCriticalSection(&FileObject->Lock); //SwitchToThread() EnterCriticalSection(&FileObject->Lock); LeaveCriticalSection(&FileObject->Lock); //SwitchToThread() EnterCriticalSection(&FileObject->Lock); LeaveCriticalSection(&cs); LeaveCriticalSection(&FileObject->Lock); try it execute in concurent in 2 threads: thread_1 thread_2 --------------------------------------------------------------------------------- ----- EnterCriticalSection(&FileObject->Lock); EnterCriticalSection(&cs); LeaveCriticalSection(&FileObject->Lock); EnterCriticalSection(&FileObject->Lock); EnterCriticalSection(&cs); EnterCriticalSection(&FileObject->Lock); also note if thread_1 will be use FileObject1->Lock and thread_2 will be use FileObject2->Lock and only cs will be common - deadlock is gone. this is even not feature. need be very accurate when we sequential acquire two critical sections
Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You must login to OSR Online AND be a member of the ntdev list to be able to post.

All times are GMT -5. The time now is 06:51.


Copyright ©2015, OSR Open Systems Resources, Inc.
Based on vBulletin Copyright ©2000 - 2005, Jelsoft Enterprises Ltd.
Modified under license