OSRLogo
OSRLogoOSRLogoOSRLogo x Subscribe to The NT Insider
OSRLogo
x

Everything Windows Driver Development

x
x
x
GoToHomePage xLoginx
 
 

    Wed, 13 Dec 2017     115635 members

   Login
   Join


 
 
Contents
  Online Dump Analyzer
OSR Dev Blog
The NT Insider
Downloads
ListServer / Forum
Driver Jobs
  Express Links
  · The NT Insider Digital Edition - May-June 2016 Now Available!
  · Windows 8.1 Update: VS Express Now Supported
  · HCK Client install on Windows N versions
  · There's a WDFSTRING?
  · When CAN You Call WdfIoQueueP...ously

Investigating a NULL Pointer Dereference

The following is an analysis that I wrote up based on a crash dump provided by a former Kernel Debugging and Crash Analysis for Windows student. Thought that the information might be of use to others and that I'd share it here.

 

2: kd> !analyze -v

*******************************************************************************

* *

* Bugcheck Analysis *

* *

*******************************************************************************

 

KERNEL_MODE_EXCEPTION_NOT_HANDLED (8e)

This is a very common bugcheck. Usually the exception address pinpoints

the driver/function that caused the problem. Always note this address

as well as the link date of the driver/image that contains this address.

Some common problems are exception code 0x80000003. This means a hard

coded breakpoint or assertion was hit, but this system was booted

/NODEBUG. This is not supposed to happen as developers should never have

hardcoded breakpoints in retail code, but ...

If this happens, make sure a debugger gets connected, and the

system is booted /DEBUG.? This will let us see why this breakpoint is

happening.

Arguments:

Arg1: c0000005, The exception code that was not handled

Arg2: 9dc8f09d, The address that the exception occurred at

Arg3: 93dc3964, Trap Frame

Arg4: 00000000

 

Debugging Details:

------------------

 

 

FAULTING_IP:

win32k!xxxProcessEventMessage+36e

9dc8f09d mov edx,dword ptr [edx]

 

TRAP_FRAME:? 93dc3964 -- (.trap 0xffffffff93dc3964)

ErrCode = 00000000

eax=fe863cd0 ebx=88c68048 ecx=00004110 edx=00000000 esi=93dc3aac edi=fe4ba860

eip=9dc8f09d esp=93dc39d8 ebp=93dc39fc iopl=0???????? nv up ei pl nz na pe nc

cs=0008? ss=0010? ds=0023? es=0023? fs=0030? gs=0000???????????? efl=00010206

win32k!xxxProcessEventMessage+0x36e:

9dc8f09d mov edx,dword ptr [edx] ds:0023:00000000=????????

Resetting default scope

 

DEFAULT_BUCKET_ID: WIN7_DRIVER_FAULT

 

BUGCHECK_STR: 0x8E

 

PROCESS_NAME: AcroRd32.exe

 

CURRENT_IRQL: 0

 

LAST_CONTROL_TRANSFER: from 82eed68e to 82f17e98

 

STACK_TEXT:

93dc34d4 82eed68e 0000008e c0000005 9dc8f09d nt!KeBugCheckEx+0x1e

93dc38f4 82e774e6 93dc3910 00000000 93dc3964 nt!KiDispatchException+0x1ac

93dc395c 82e7749a 93dc39fc 9dc8f09d badb0d00 nt!CommonDispatchException+0x4a

93dc3988 9dcacbc7 00000000 00000000 00000023 nt!Kei386EoiHelper+0x192

93dc39fc 9dcb51c9 fe4ba860 fe042458 0e3effc3 win32k!PostInputMessage+0x126

93dc3b4c 9dce7f50 fe4ba860 93dc3be4 00000000 win32k!xxxScanSysQueue+0x291

93dc3bb4 9dcdcf0e 93dc3be4 000025ff 00000000 win32k!xxxRealInternalGetMessage+0x32e

93dc3c18 82e768fa 002efca0 00000000 00000000 win32k!NtUserPeekMessage+0x3f

93dc3c18 77d97094 002efca0 00000000 00000000 nt!KiFastCallEntry+0x12a

WARNING: Frame IP not in any known module. Following frames may be wrong.

002efc40 00000000 00000000 00000000 00000000 0x77d97094

 

The bugcheck is a vanilla NULL pointer deref, where EDX contains the NULL value:

 

2: kd> r

Last set context:

eax=fe863cd0 ebx=88c68048 ecx=00004110 edx=00000000 esi=93dc3aac edi=fe4ba860

eip=9dc8f09d esp=93dc39d8 ebp=93dc39fc iopl=0         nv up ei pl nz na pe nc

cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010206

win32k!xxxProcessEventMessage+0x36e:

9dc8f09d 8b12            mov     edx,dword ptr [edx]  ds:0023:00000000=????????

 

In the interest of finding out where EDX came from, unassemble the faulting function:

 

2: kd> uf win32k!xxxProcessEventMessage

Flow analysis was incomplete, some code may be missing

win32k!xxxProcessEventMessage:

9dc8ed2f mov     edi,edi

9dc8ed31 push    ebp

9dc8ed32 mov     ebp,esp

9dc8ed34 sub     esp,18h

9dc8ed37 push    esi

9dc8ed38 mov     esi,dword ptr [ebp+0Ch]

9dc8ed3b push    edi

9dc8ed3c push    offset win32k!CleanEventMessage (9dca6f32)

9dc8ed41 lea     eax,[ebp-18h]

9dc8ed44 push    eax

9dc8ed45 push    esi

9dc8ed46 call    win32k!PushW32ThreadLock (9dcf3483)

9dc8ed4b mov     eax,dword ptr [esi+30h]

9dc8ed4e mov     edi,dword ptr [ebp+8]

9dc8ed51 mov     ecx,dword ptr [edi+0BCh]

9dc8ed57 and     eax,3FFFFFFFh

9dc8ed5c dec     eax

9dc8ed5d mov     dword ptr [ebp+0Ch],ecx

9dc8ed60 cmp     eax,10h

9dc8ed63 ja      win32k!xxxProcessEventMessage+0x567 (9dc8f296)

 

win32k!xxxProcessEventMessage+0x3a:

9dc8ed69 push    ebx

9dc8ed6a jmp     dword ptr win32k!xxxProcessEventMessage+0x578 (9dc8f2a5)[eax*4]

 

win32k!xxxProcessEventMessage+0x567:

9dc8f296 lea     eax,[ebp-18h]

9dc8f299 push    eax

9dc8f29a call    win32k!PopW32ThreadLock (9dca90fc)

9dc8f29f pop     edi

9dc8f2a0 pop     esi

9dc8f2a1 leave

9dc8f2a2 ret     8

 

 

Pretty short function, but our faulting instruction is not included! The trouble is this line:

 

9dc8ed6a jmp     dword ptr win32k!xxxProcessEventMessage+0x578 (9dc8f2a5)[eax*4]

 

We likely have jumped through this instruction, which WinDbg isn't following with the uf command. The trouble is that the path taken depends on the value of EAX at the point of the call sequence. If we dump this area out with dps we should expect to see an array of function pointers:

 

2: kd> dps 9dc8f2a5

9dc8f2a5  9dc8edb8 win32k!xxxProcessEventMessage+0x89

9dc8f2a9  9dc8efd7 win32k!xxxProcessEventMessage+0x2a8

9dc8f2ad  9dc8ee23 win32k!xxxProcessEventMessage+0xf4

9dc8f2b1  9dc8ee30 win32k!xxxProcessEventMessage+0x101

9dc8f2b5  9dc8efc9 win32k!xxxProcessEventMessage+0x29a

9dc8f2b9  9dc8ee43 win32k!xxxProcessEventMessage+0x114

9dc8f2bd  9dc8f019 win32k!xxxProcessEventMessage+0x2ea

9dc8f2c1  9dc8ed71 win32k!xxxProcessEventMessage+0x42

9dc8f2c5  9dc8f03f win32k!xxxProcessEventMessage+0x310

9dc8f2c9  9dc8f04c win32k!xxxProcessEventMessage+0x31d

9dc8f2cd  9dc8f07f win32k!xxxProcessEventMessage+0x350

9dc8f2d1  9dc8ee16 win32k!xxxProcessEventMessage+0xe7

9dc8f2d5  9dc8f138 win32k!xxxProcessEventMessage+0x409

9dc8f2d9  9dc8f16d win32k!xxxProcessEventMessage+0x43e

9dc8f2dd  9dc8f1aa win32k!xxxProcessEventMessage+0x47b

9dc8f2e1  9dc8f22b win32k!xxxProcessEventMessage+0x4fc

9dc8f2e5  9dc8f261 win32k!xxxProcessEventMessage+0x532

9dc8f2e9  90909090

9dc8f2ed  55ff8b90

9dc8f2f1  ec83ec8b

9dc8f2f5  1d8b5350

9dc8f2f9  9de43db8 win32k!gptiCurrent

9dc8f2fd  08758b56

9dc8f301  00bc868b

9dc8f305  33570000

9dc8f309  fc5d89ff

9dc8f30d  39f87d89

9dc8f311  840f2878

9dc8f315  000002a0

9dc8f319  0a74f33b

9dc8f31d  50b0458d

9dc8f321  cf34e856

 

Unfortunately it's more than one, so to figure out the code sequence we either need to:

 

1.       Go through these one by one with the u/uf command until we find one that leads to our sequence

2.       Reconstruct EAX at the time of the crash

 

I tried #1 first and quickly gave up. So, back to the disassembly of the function with just the interesting pieces shown:

 

win32k!xxxProcessEventMessage:

9dc8ed2f mov     edi,edi

9dc8ed31 push    ebp

9dc8ed32 mov     ebp,esp

...

9dc8ed38 mov     esi,dword ptr [ebp+0Ch]

...

9dc8ed4b mov     eax,dword ptr [esi+30h]

...

9dc8ed57 and     eax,3FFFFFFFh

9dc8ed5c dec     eax

9dc8ed5d mov     dword ptr [ebp+0Ch],ecx

9dc8ed60 cmp     eax,10h

9dc8ed63 ja      win32k!xxxProcessEventMessage+0x567 (9dc8f296)

 

win32k!xxxProcessEventMessage+0x3a:

9dc8ed69 push    ebx

9dc8ed6a jmp     dword ptr win32k!xxxProcessEventMessage+0x578 (9dc8f2a5)[eax*4]

 

Thus, the path of EAX is:

 

1.       [EBP+C] goes into ESI

2.       [ESI+30] goes into EAX

3.       Mask off the high two bits of EAX (EAX & 0x3FFFFFFF)

4.       Decrement EAX by one

 

Thus, in order to reconstruct we either need [EBP+C] or ESI. You'll note however that [EBP+C] is overwritten as part of this sequence (highlighted in red), so we can?t get the correct value of [EBP+C] from this frame. However, we're in an EBP frame and therefore we know that EBP+C is the second stack passed parameter. Therefore, we should be able to reconstruct the value based on information in the caller's frame:

 

9dcb51b8 and     dword ptr [eax+14h],0

9dcb51bc lea     eax,[ebp-0A0h]

9dcb51c2 push    eax

9dcb51c3 push    ebx

9dcb51c4 call    win32k!xxxProcessEventMessage (9dc8ed2f)

9dcb51c9 jmp     win32k!xxxScanSysQueue+0xf2 (9dcb502a)

 

If this is correct, we should be able to figure out the second stack passed argument: it's the caller's EBP-A0 (note that there?s an LEA instruction here, so no pointer deref!).

 

2: kd> r

Last set context:

eax=fe863cd0 ebx=88c68048 ecx=00004110 edx=00000000 esi=93dc3aac edi=fe4ba860

eip=9dc8f09d esp=93dc39d8 ebp=93dc39fc iopl=0         nv up ei pl nz na pe nc

cs=0008  ss=0010  ds=0023  es=0023  fs=0030  gs=0000             efl=00010206

win32k!xxxProcessEventMessage+0x36e:

9dc8f09d mov     edx,dword ptr [edx]                  ds:0023:00000000=????????

 

2: kd> * Use our EBP to get the caller's EBP

2: kd> dd @ebp l1

93dc39fc  93dc3b4c

 

2: kd> * Subtract A0 to get the stack passed argument

2: kd> ? @$p-A0

Evaluate expression: -1814283604 = 93dc3aac

 

2: kd> * This +30 should give us our initial EAX value:

2: kd> dd 93dc3aac+30 l1

93dc3adc  0000000b

 

2: kd> * AND it with 3FFFFFFF and subtract one

2: kd> ? (@$p & 0x3FFFFFFF) - 1

Evaluate expression: 10 = 0000000a

 

 

We should then expect to see our faulting sequence located at index A of the jump table:

 

win32k!xxxProcessEventMessage+0x3a:

9dc8ed69 push    ebx

9dc8ed6a jmp     dword ptr win32k!xxxProcessEventMessage+0x578 (9dc8f2a5)[eax*4]

 

2: kd> dps 9dc8f2a5 + (a*4) l1

9dc8f2cd  9dc8f07f win32k!xxxProcessEventMessage+0x350

2: kd> u win32k!xxxProcessEventMessage+0x350

win32k!xxxProcessEventMessage+0x350:

9dc8f07f mov     ecx,dword ptr [esi+8]

9dc8f082 mov     ebx,dword ptr [edi+0C8h]

9dc8f088 mov     dl,1

9dc8f08a call    win32k!HMValidateHandleNoSecure (9dce8623)

9dc8f08f mov     ecx,dword ptr [ebx+14h]

9dc8f092 test    ecx,5C00h

9dc8f098 je      win32k!xxxProcessEventMessage+0x381 (9dc8f0b0)

9dc8f09a mov     edx,dword ptr [ebx+64h]

9dc8f09d mov     edx,dword ptr [edx]

 

That's great and all, but given that what do we know?

 

Well, we can now see that EDX came from EBX+64:

 

2: kd> dd @ebx+64 l1

88c680ac  00000000

 

And this means we can try to see if EBX is anything interesting:

 

2: kd> !pool @ebx 1

Pool page 88c68048 region is Nonpaged pool

*88c68000 size:   d0 previous size:    0  (Allocated) *Desk (Protected)

              Owning component : Unknown (update pooltag.txt)

    88c68008  00000000 00000084 00000070 82f6fa00

    88c68018  e38412c8 00000001 00000000 0010000e

    88c68028  8fa7c558 00000000 00000e74 00000027

    88c68038  00000000 000e0015 82f6fa00 bc1f3855

    88c68048  00000001 fe800578 ff9d1728 88c5bf78

    88c68058  88c6bd28 00004110 80480004 fe802a68

    88c68068  fe80ee70 00000000 00000000 00000000

    88c68078  fe87da48 fe800748 fe800880 8fa87fd8

    88c68088  fe800000 00c00000 000604e8 000001a0

    88c68098  000003fc 000001a8 00000408 fd053ac0

    88c680a8  ff553960 00000000 00050045 00000744

    88c680b8  0000003b 00000748 0000003f 00000190

    88c680c8  00000000 000e082c

 

 

 

NULL pointer is highlighted in green in the output. Just based on the dump of the memory region, I can't immediately characterize this as a corruption in any kind of meaningful way (i.e. the entire structure isn't zero or anything).

 

I suspect that this allocation represents a desktop object just based on the tag, "Desk". This is an object used by the graphics subsystem to indicate, well, a desktop. I can confirm my guess by using !object:

 

2: kd> !object @ebx

Object: 88c68048  Type: (856c89c8) Desktop

    ObjectHeader: 88c68030 (new version)

    HandleCount: 39  PointerCount: 3700

    Directory Object: 00000000  Name: Default

 

Given all of that, it all looks pretty in order. We have a NULL pointer deref of a common graphics subsystem object, I don't see any obvious third party interference. Just because it's a NULL pointer, I decided to look at the memory state of the machine:

 

2: kd> !vm 21

 

*** Virtual Memory Usage ***

       Physical Memory:      894268 (   3577072 Kb)

       Page File: \??\C:\pagefile.sys

         Current:   3577072 Kb  Free Space:   1380964 Kb

         Minimum:   3577072 Kb  Maximum:     10731216 Kb

       Available Pages:      225970 (    903880 Kb)

       ResAvail Pages:       610095 (   2440380 Kb)

       Locked IO Pages:           0 (         0 Kb)

       Free System PTEs:      36039 (    144156 Kb)

       Modified Pages:         5104 (     20416 Kb)

       Modified PF Pages:      4673 (     18692 Kb)

       NonPagedPool Usage:    24142 (     96568 Kb)

       NonPagedPool Max:     522760 (   2091040 Kb)

       PagedPool 0 Usage:     55224 (    220896 Kb)

       PagedPool 1 Usage:     10576 (     42304 Kb)

       PagedPool 2 Usage:      7263 (     29052 Kb)

       PagedPool 3 Usage:      7277 (     29108 Kb)

       PagedPool 4 Usage:      7390 (     29560 Kb)

       PagedPool Usage:       87730 (    350920 Kb)

       PagedPool Maximum:    523264 (   2093056 Kb)

 

       ********** 11 pool allocations have failed **********

 

       Session Commit:        14541 (     58164 Kb)

       Shared Commit:        220028 (    880112 Kb)

       Special Pool:              0 (         0 Kb)

       Shared Process:         4482 (     17928 Kb)

       PagedPool Commit:      87788 (    351152 Kb)

       Driver Commit:         19638 (     78552 Kb)

       Committed pages:     1280380 (   5121520 Kb)

       Commit limit:        1788098 (   7152392 Kb)

 

 

       VA Type           CurrentUse  Peak      Limit          Failures

       Unused              134 Mb       0 Mb   OPEN              0

       SessionSpace         68 Mb     106 Mb   OPEN              4

       ProcessSpace         16 Mb       0 Mb   OPEN              0

       BootLoaded           52 Mb      66 Mb   OPEN              0

       PfnDatabase          22 Mb     128 Mb   OPEN              0

       NonPagedPool        110 Mb     114 Mb   OPEN              0

       PagedPool           610 Mb     618 Mb   OPEN             18

       SpecialPool           0 Mb       0 Mb   OPEN              0

       SystemCache         648 Mb    1026 Mb   OPEN             54

       SystemPtes          322 Mb     346 Mb   OPEN              1

       Hal                   4 Mb       0 Mb   OPEN              0

       SessionGlobal        12 Mb      12 Mb   OPEN              0

       Driver Images        50 Mb       0 Mb   OPEN              0

       NPSpecialPool         0 Mb       0 Mb   OPEN              0

       ProtoPTE Pool         0 Mb       0 Mb   OPEN              0

 

                Maximum contiguous unused VA:   10 Mb

 

 

Pretty suspicious that we have some memory allocation failures. However, I also see that the system has plenty of free memory at the moment so these could just be red herrings.

 

Lastly, I used !sysinfo to figure out what type of computer this is:

 

2: kd> !sysinfo smbios

[SMBIOS Data Tables v2.6]

[DMI Version - 38]

[2.0 Calling Convention - No]

[Table Size - 2448 bytes]

...

  Family                        ThinkPad T420

...

 

And did some quick Googling to see if this is a known issue. Unfortunately, I came up with nada, so another dead end.

 

If I were really on the spot to figure this out, I'd probably set a breakpoint in this code path to walk through the working case. From there, I could figure out what the NULL value is supposed to be and start trying to think of how it might have become NULL.

 

 

Related Articles
Enabling Debugging on the Local Machine for Windows XP®
You're Testing Me - Testing WDM/Win2K Drivers
Analyze This - Analyzing a Crash Dump
More on Kernel Debugging - KMODE_EXCEPTION_NOT_HANDLED
Making WinDbg Your Friend - Creating Debugger Extensions
Life Support for WinDbg - New Windows NT Support Tools
Life After Death? - Understanding Blue Screens
Special Win2K PnP Tracing and Checks
All About Lint - PC Lint and Windows Drivers
Bagging Bugs — Avoidance and Detection Tips to Consider

Writing WDF Drivers I: Core Concepts
LAB

Nashua (Amherst), NH
15-19 May 2017

Writing WDF Drivers II: Advanced Implementation Techniques
LAB

Nashua (Amherst), NH
23-26 May 2017

Kernel Debugging and Crash Analysis
LAB

Dulles (Sterling), VA
26-30 Jun 2017

Windows Internals and Software Driver Development
LAB

Nashua (Amherst), NH
24-28 Jul 2017

 
 
 
 
x
LetUsHelp
 

Need to develop a Windows file system solution?

We've got a kit for that.

Need Windows internals or kernel driver expertise?

Bring us your most challenging project - we can help!

System hangs/crashes?

We've got a special diagnostic team that's standing by.

Visit the OSR Corporate Web site for more information about how OSR can help!

 
bottom nav links