Getting Crash Dumps to Appear in Win7
The NT Insider, Volume 16, Issue 4, Nov-Dec 2009 | Published: 09-Dec-09| Modified: 09-Dec-09
With everyone eagerly installing Windows 7 on their systems here at OSR, a couple of interesting crash dump mysteries were brought to light. The first mystery turns out to be a remnant of an O/S long forgotten whereas the second mystery represents a new policy regarding Windows' retention of crash dump files.
Windows 7 No Longer Supports Full Memory Dumps??
Did that catch your attention? It certainly caught mine and became a hot topic of conversation at lunch one day. What spurred this comment was the disappearance of the "Complete memory dump" option from the debugging information dialog on a laptop running Windows 7 x86:
The initial collective guess was that the paging file was not large enough on this system to hold a full memory dump, given that the system had 4GB of memory installed. Unfortunately, the paging file on this system was more than large enough so we continued to be stumped.
Looking at another system on the network, we discovered an XP x64 workstation with 8GB of RAM that was also missing the complete memory dump option. Then we discovered a Windows 7 VM with 2GB of RAM that did have the complete memory dump option. Clearly there was something going on here that we didn't quite understand, which meant that this was now officially a challenge.
Dialogs Can Be Deceiving
After spending much time making guesses as to what would cause the O/S to show or not show the complete memory dump option, we finally sat down and started digging into the control panel applet responsible for this dialog, sysdm.cpl. This unveiled an interesting code path, as can be seen in this IDA output:
Notice that this performs a comparison of a variable with 2GB, jumping over the addition of a string from the image's string table to a combo box if the variable value is above 2GB. Loading the string table of the control panel applet, we found that string 0x4E was "Complete memory dump".
Suspicious that the 2GB value was a compare of the amount of RAM installed, we went back to check the amount of memory that was in each system and whether or not the complete memory dump option was available:
1) Windows 7 x86 laptop - 4GB and no option
2) Windows XP x64 workstation - 8GB and no option
3) Windows 7 x64 VM - 2GB and had the option
We then went ahead and added another 512MB of RAM to the Windows 7 VM. As expected, the complete memory dump option disappeared on the next boot!
Knowing for a fact that Windows supported crash dumps larger than 2GB, we took to Google to finally solve this mystery. As it turns out, the I/O Manager originally limited the size of a complete memory dump to 2GB, but that check was removed via a hotfix to Windows 2000. While the hotfix updated the I/O Manager to remove the check, no one has ever bothered to go back and update the control panel applet that displays the crash dump registry settings.
Luckily, the workaround for this is easy enough. You'll need to bypass the dialog and manually modify the registry yourself, setting the \HKLM\System\CCS\Control\ CrashControl\CrashDumpEnabled DWORD value to 1. By doing so you'll configure the O/S to generate a full memory dump on the next system crash and you'll even regain access to the complete memory dump option in the dialog:
That's Great, But Where's My Crash Dump?
Now that that mystery was solved, there was even a greater mystery brewing: Windows 7 systems that were configured to generate crash dumps were crashing but not leaving behind a dump file. After having ruled out the usual reasons such as too small of a paging file or not enough disk space to save the dump, there was definitely something strange going on.
What we discovered is that a new policy has been added to Windows when it comes to crash dump file retention. Apparently, certain customers are complaining about the amount of disk space that crash dump files consume and don't want them saved anymore (we're sure they'll regret this decision in the future if their systems ever crash). In order to address this issue, a new set of criteria has been created that determines whether or not Windows saves a crash dump file.
The new criteria are as follows:
1) If the machine is running a server version of the O/S, the crash dump is always saved.
2) If the machine is joined to a domain, the crash dump is always saved.
3) If the amount of free space on the disk is greater than or equal to 25GB, the crash dump is saved.
Otherwise, the crash dump file will not be saved and you'll be left scratching your head. Luckily, a new registry value has been added that will override this behavior and always cause the crash dump file to be generated. By setting the \HKLM\System\CCS\Control\CrashControl\AlwaysKeepMemoryDump DWORD value to 1 you will guarantee that you will always have a crash dump file after crashing the system.
More Mysteries to Come?
A new release of the O/S is always time to discover old issues that you never noticed before and experience new issues that never existed before. As adoption for Windows 7 grows I'm sure there will be more articles like this to come in future issues of The NT Insider!
Analyze This - Analyzing a Crash Dump
More on Kernel Debugging - KMODE_EXCEPTION_NOT_HANDLED
Life After Death? - Understanding Blue Screens
Wild Speculation -- Debugging Another Crash Dump
Finding File Contents in Memory
Inlining into SEH Filters Can Result in Invalid Code on AMD64
Debugging 103: Where To Go With A System Crash
Investigating a NULL Pointer Dereference
Rate this article and give us feedback. Do you find anything missing? Share your opinion with the community!
Post Your Comment
"Dump storage and cleanup behavior has been changed."
You can see this feature below document. The policy has been changed. http://blogs.msdn.com/wer/archive/2009/02/09/kernel-dump-storage-and-clean-up-behavior-in-windows-7.aspx
18-Dec-09, taehwa lee