Does the dump header change?

NTDEV
1

I have a tool that converts xen dump-core memory dumps obtained from Dom0 to WinDBG debuggable dumps. It works by skipping the ‘elf’ header, then dropping a range of pages, and finally pre-pending a crash dump header that is obtained at boot time by a call to KeInitializeCrashDumpHeader. This results in a .dmp file that is nearly identical to one that is obtained when the system dumps properly. In the past, I’ve used this to debug VMs that crash and don’t yield a dump (often due to a fault in the storage stack).

I received a report that the tool wasn’t working with a Win2k3 x64 VM, so I commandeered the machine and used ctrl-scroll-scroll to force a crash that would produce a dump. I then produced a xen dump-core dump and ran it through my tool. Sure enough, WinDBG refused to open the file (The file is corrupt or invalid). In the end, if I took the first two pages from the REAL .dmp file and replaced that of the manufactured one, it would work.

An older version of my driver would refresh the dump header at crash time with a bugcheck callback, but I found that at some point I had introduced a bug in my code such that the header is not refreshed at bugcheck time. Is this fatal? Does the contents of the (would be) dump header change while the OS runs?

2

>Does the contents of the (would be) dump header change while the OS runs?

Information in the header changes based on the thread that you’re in when
the system crashes (i.e. directory table base info and context record info),
though that’s likely to be unrelated to this really. The only thing I can
think of would be if you dynamically added RAM to the VM?

The docs on the API are pretty clear that it does *not* need to be called
immediately before the generation of the dump, so sounds like something else
is up.

-scott

–
Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntdev…

I have a tool that converts xen dump-core memory dumps obtained from Dom0 to
WinDBG debuggable dumps. It works by skipping the ‘elf’ header, then
dropping a range of pages, and finally pre-pending a crash dump header that
is obtained at boot time by a call to KeInitializeCrashDumpHeader. This
results in a .dmp file that is nearly identical to one that is obtained when
the system dumps properly. In the past, I’ve used this to debug VMs that
crash and don’t yield a dump (often due to a fault in the storage stack).

I received a report that the tool wasn’t working with a Win2k3 x64 VM, so I
commandeered the machine and used ctrl-scroll-scroll to force a crash that
would produce a dump. I then produced a xen dump-core dump and ran it
through my tool. Sure enough, WinDBG refused to open the file (The file is
corrupt or invalid). In the end, if I took the first two pages from the
REAL .dmp file and replaced that of the manufactured one, it would work.

An older version of my driver would refresh the dump header at crash time
with a bugcheck callback, but I found that at some point I had introduced a
bug in my code such that the header is not refreshed at bugcheck time. Is
this fatal? Does the contents of the (would be) dump header change while
the OS runs?

3

The value of “DirectoryTableBase” is definitely different (a field that DumpChk.exe dumps out when you give it a good .dmp) - although I don’t know if this is relevant to a valid dump.

4

Probably just the fact that it’s present and a valid directory table base is
enough.

Too bad that WinDBG won’t open it at all, that makes it a bit trickier to
track down what it doesn’t like. You could always build a copy of dumpstk
from the WinDBG SDK and see if that will open it. If not, you’ll at least
have an easier time finding a starting point in DbgEng to walk through to
find the spot where the crash dump open bails.

Another option would be to parse the dump header yourself and compare the
two to figure out which fields are different. I don’t believe that the
structure is documented, but .dumpdebug on the good crash dump should give
you a good starting point (or you might find the structure online).

-scott

–
Scott Noone
Consulting Associate and Chief System Problem Analyst
OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntdev…

The value of “DirectoryTableBase” is definitely different (a field that
DumpChk.exe dumps out when you give it a good .dmp) - although I don’t know
if this is relevant to a valid dump.

5

If WinDbg fails to open the crash dump, it’s almost certainly dying somewhere in DbgEng (i.e. you’re likely to see an identical failure mode). Dumpchk.exe, which ships with the debugger, is a minimalistic DbgEng shell that runs a few precanned commands after opening a dump and exits; if you want a simplified target to debug, I would use that.

Otherwise, Scott’s suggestion is probably the best one - diff the dump header fields, find out which are different, then match the field values up with the .dumpdebug output to see what the fields mean.

If you really want to step through the debugger loading of the crash dump, dbgeng!KernelFull64DumpTargetInfo::InitializeFile is probably the place you’d want to start with a breakpoint if memory serves (for an NT64 kernel full dump a-la what KeInitializeCrashDumpHeader will set you up on NT64). But comparing the fields is likely to prove less painful.

  • S

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of Scott Noone
Sent: Wednesday, April 20, 2011 9:47 AM
To: Windows System Software Devs Interest List
Subject: Re:[ntdev] Does the dump header change?

Probably just the fact that it’s present and a valid directory table base is enough.

Too bad that WinDBG won’t open it at all, that makes it a bit trickier to track down what it doesn’t like. You could always build a copy of dumpstk from the WinDBG SDK and see if that will open it. If not, you’ll at least have an easier time finding a starting point in DbgEng to walk through to find the spot where the crash dump open bails.

Another option would be to parse the dump header yourself and compare the two to figure out which fields are different. I don’t believe that the structure is documented, but .dumpdebug on the good crash dump should give you a good starting point (or you might find the structure online).

-scott

–
Scott Noone
Consulting Associate and Chief System Problem Analyst OSR Open Systems Resources, Inc.
http://www.osronline.com

wrote in message news:xxxxx@ntdev…

The value of “DirectoryTableBase” is definitely different (a field that DumpChk.exe dumps out when you give it a good .dmp) - although I don’t know if this is relevant to a valid dump.

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

6

I did :slight_smile: Output below… None of the ‘diffs’ seem to be addresses that show up in the DumpChk output, so I’m assuming they pertain to the crash. Thank you guys for the other ideas though - I have new things to try out from both posts…

G:\dave\Dave3>bincomp MEMORY.DMP new.dmp
00000011: 60 10
00000012: 79 ec
00000013: 54 7e
0000001a: 40 20
000003d8: c0 b0
000003d9: 16 aa
000003da: 24 79
000003db: cd f2
000003e0: b8 e8
000003e1: 08 e7
000003e2: 29 54
000003e3: c5 c6
000003e8: 00 80
000003e9: 17 00
000003ea: 29 00
000003eb: c5 00
000003ec: df 00
000003ed: fa 00
000003ee: ff 00
000003ef: ff 00
000003f0: c0 b0
000003f1: cd aa
000003f2: 46 79
000003f3: aa f2
000003f4: bd df
000003f8: 60 c0
000003f9: e0 3f
000003fa: 22 e2
000003fb: ce f1
00000420: 00 03
00000421: 00 01
00000430: c0 40
00000431: cd 80
00000432: 46 72
00000433: aa ce
00000434: bd df
00000439: cf d6
0000043a: 46 87
0000043b: aa c8
0000043c: bd df
000004e8: 00 78
000004ea: 00 f8
000004f0: 00 20
000004f1: 00 f0
000004f2: 00 80
000004f3: 00 02
000004f4: 00 80
000004f5: 00 fa
000004f6: 00 ff
000004f7: 00 ff
00000f98: 01 02
00000fa1: 00 50
00000fa2: e0 62
00000fa3: 7f 28
00000fa8: 60 74
00000fa9: a3 ee
00000faa: e5 d7
00000fab: 37 90
00000fac: dc df
00000fad: fd fa
00001030: e6 66
00001031: 63 9e
00001032: 1a d9
00001033: 86 0d
00001034: 01 e6

C:\Program Files\Support Tools>dumpchk g:\dave\memory.dmp
Loading dump file g:\dave\memory.dmp
----- 64 bit Kernel Full Dump Analysis

DUMP_HEADER64:
MajorVersion 0000000f
MinorVersion 00000ece
DirectoryTableBase 0000000054796000 PfnDataBase fffffadfc9400000
PsLoadedModuleList fffff800011d4140 PsActiveProcessHead fffff800011d0e50
MachineImageType 00008664
NumberProcessors 00000002
BugCheckCode 86427531
BugCheckParameter1 0000000000000000 BugCheckParameter2 0000000000000000
BugCheckParameter3 0000000000000000 BugCheckParameter4 0000000000000000
KdDebuggerDataBlock fffff800`011aed40

7

The most interesting change here seems to me to be the directory table base (although it’s not surprising that it would be different). In that vein, in what process context were you calling KeInitializeCrashDumpHeader? Was the call made in the context of the system process, or some other (user) process that might have gone away by the time the machine crashed?

  • S (Msft)

-----Original Message-----
From: xxxxx@lists.osr.com [mailto:xxxxx@lists.osr.com] On Behalf Of xxxxx@oracle.com
Sent: Wednesday, April 20, 2011 10:37 AM
To: Windows System Software Devs Interest List
Subject: RE:[ntdev] Does the dump header change?

I did :slight_smile: Output below… None of the ‘diffs’ seem to be addresses that show up in the DumpChk output, so I’m assuming they pertain to the crash. Thank you guys for the other ideas though - I have new things to try out from both posts…

G:\dave\Dave3>bincomp MEMORY.DMP new.dmp
00000011: 60 10
00000012: 79 ec
00000013: 54 7e
0000001a: 40 20
000003d8: c0 b0
000003d9: 16 aa
000003da: 24 79
000003db: cd f2
000003e0: b8 e8
000003e1: 08 e7
000003e2: 29 54
000003e3: c5 c6
000003e8: 00 80
000003e9: 17 00
000003ea: 29 00
000003eb: c5 00
000003ec: df 00
000003ed: fa 00
000003ee: ff 00
000003ef: ff 00
000003f0: c0 b0
000003f1: cd aa
000003f2: 46 79
000003f3: aa f2
000003f4: bd df
000003f8: 60 c0
000003f9: e0 3f
000003fa: 22 e2
000003fb: ce f1
00000420: 00 03
00000421: 00 01
00000430: c0 40
00000431: cd 80
00000432: 46 72
00000433: aa ce
00000434: bd df
00000439: cf d6
0000043a: 46 87
0000043b: aa c8
0000043c: bd df
000004e8: 00 78
000004ea: 00 f8
000004f0: 00 20
000004f1: 00 f0
000004f2: 00 80
000004f3: 00 02
000004f4: 00 80
000004f5: 00 fa
000004f6: 00 ff
000004f7: 00 ff
00000f98: 01 02
00000fa1: 00 50
00000fa2: e0 62
00000fa3: 7f 28
00000fa8: 60 74
00000fa9: a3 ee
00000faa: e5 d7
00000fab: 37 90
00000fac: dc df
00000fad: fd fa
00001030: e6 66
00001031: 63 9e
00001032: 1a d9
00001033: 86 0d
00001034: 01 e6

C:\Program Files\Support Tools>dumpchk g:\dave\memory.dmp Loading dump file g:\dave\memory.dmp
----- 64 bit Kernel Full Dump Analysis

DUMP_HEADER64:
MajorVersion 0000000f
MinorVersion 00000ece
DirectoryTableBase 0000000054796000 PfnDataBase fffffadfc9400000
PsLoadedModuleList fffff800011d4140 PsActiveProcessHead fffff800011d0e50
MachineImageType 00008664
NumberProcessors 00000002
BugCheckCode 86427531
BugCheckParameter1 0000000000000000 BugCheckParameter2 0000000000000000
BugCheckParameter3 0000000000000000 BugCheckParameter4 0000000000000000
KdDebuggerDataBlock fffff800`011aed40

NTDEV is sponsored by OSR

For our schedule of WDF, WDM, debugging and other seminars visit:
http://www.osr.com/seminars

To unsubscribe, visit the List Server section of OSR Online at http://www.osronline.com/page.cfm?name=ListServer

8

If I’m right, and the code to refresh the crashdumpheader during bugcheck time was buggy and failed, then that crash dump header was obtained during AddDevice of the bus driver.

The bug was that, at AddDevice time, I’m supposed to test for the existence of KeInitializeCrashDumpHeader (Win2k3 or later), and allocate the 2 page buffer for it. Then at crash time, I’m only supposed to call KeInitializeCrashDumpHeader, but I was instead calling the setup function again, so i’m assuming the MmGetSystemRoutineAddress and/or ExAllocatePoolWithTag failed and the dump header was not refreshed.

I’ll take the manufactured dump and ONLY replace the 3 bytes of diff that represent DirectoryTableBase so we can be sure if that is the critical diff… I wish the binary editor in visual studio was smart enough to not load entire files into memory when we open them…

9

>I wish the binary editor in visual studio was smart enough …

Then again, I wish I was smart enough to not have invented the dumpconverter!

I was sure I had set someone up for that slam dunk… You guys must be distracted today.