By Ralph Shnelvar and Jamey Kirby

[Status Update: As of 22 March 2005, this bug was fixed in Windows "Longhorn."  Thanks to the ms dev who took this bug and fixed it!]

Be grateful to the editors of The NT Insider that you are about to be told about a bug that is (1) quite likely to exist in your code, (2) exists in a lot of "working" code, (3) has been a bug since NT 4.0, and (4) is a bug that several (unnamed) knowledgeable people refused to believe is a bug.

Like so many other kernel bugs, it was time to connect up a 1394 cable and crank up WinDbg, put in a breakpoint at DriverEntry wrapped in a __try/__except block (so that I can run this with and without the kernel debugger running), and then step through the code to see where it crashes.

Oh dastardly expletive! This was one of those wretched Heisenbugs. (A Heisenbug is, of course, a bug that goes away when you put in debugging statements or invoke a debugger.)

The crashes kept moving around. It was never in my code. It was always elsewhere so that I couldn't even tell what was happening or how much of my code had run before the crash occurred.

For me, one of the many frustrations about kernel mode crashes is that coaxing Windows to produce a memory.dmp file is more art than science. I still can't reliably produce one. Sometimes a crash won't write a memory dump to pagefile.sys; sometimes you'll see it dumping memory in the BSOD. Sometimes the dump is done, but then the next reboot on the same partition doesn't. Sometimes it does and sometimes it doesn't. Don't ask me why.

On those rare occasions when a memory.dmp was actually produced, I used a file viewer to look for the memory trace. Okay; the crash showed up some time after I entered AddDevice. I added a lot of debugging code to validate subscripts and other parameters. Unfortunately, everything checked out--but we were still crashing when we booted without a debugger.

Time to break out a bigger hammer. Turn on all the error checking in Driver Verifier. The symptoms were the same: Everything worked if the debugger was running, randomly crashed without the debugger installed.

I kept breaking out bigger hammers. Henry Gabryjelski from Microsoft suggested that I try

verifier.exe /flags 0xFB /all

With all of the debugging tracing turned on I finally started seeing the crashes at a consistent point in my code. The crash was no longer in AddDevice but very early in my DriverEntry code. Specifically, it was a page fault while in a call to MmGetSystemRoutineAddress. I called this routine in order to get the version of the OS.

Here's a snippet of the code that crashed:

ASSERT( KeGetCurrentIrql() == PASSIVE_LEVEL );
    UNICODE_STRING functionName;
    ASSERT( KeGetCurrentIrql() == PASSIVE_LEVEL );
    RtlInitUnicodeString( &functionName, L"RtlGetVersion" );

    ULONG_PTR IoGetRemainingStackSize_ = IoGetRemainingStackSize();
    UNREFERENCED_PARAMETER(IoGetRemainingStackSize_);
    ASSERT( KeGetCurrentIrql() == PASSIVE_LEVEL );
    GetVersion =
        reinterpret_cast<long (__stdcall *)(struct _OSVERSIONINFOW *)>(MmGetSystemRoutineAddress( &functionName ));
ASSERT( KeGetCurrentIrql() == PASSIVE_LEVEL );

I was faced with a real stumper. We see from the statement before the MmGetSystemRoutineAddress that the current IRQL is 0 (PASSIVE_LEVEL). Yet the crash is a IRQL_NOT_LESS_OR_EQUAL. Huh?

Even more disturbing are two more facts. First, the instruction at the crash is:

This is a nop that only uses registers. How can it possibly crash? Second, the mov instruction is the first instruction in MmGetSystemRoutineAddress! How in all that is holy to programmers can this be happening?

Gentle reader, I'm not being fair with you. The people reading my many, many posts on the mailing list on this subject had the benefit of portions of a crash dump. I'll point out the most relevant lines of the dump:

efl=00010046
CURRENT_IRQL: ff

I'd been seeing the "CURRENT_IRQL: ff" for several weeks. It had bothered me but no one seemed to be able to tell me how the CURRENT_IRQL could be 0xFF unless I'd walked over some memory that didn't belong to me. You see, the IRQL is stored in a section of kernel memory known as the Processor Control Region (PCR). (You can display the PCR by using the !pcr WinDbg command extension.)

So let's review the conundrum. First, we're running at passive level. Second, the instruction that is faulting only uses registers and is, in fact, a nop. Third, the instruction that is failing is the first instruction in MmGetSystemRoutine Address.

You have all the clues that I did. I couldn't figure it out but OSR's very own Tony Mason did. His answer was:

Later that same week, Microsoft's Jake Oshins added a rather comprehensive explanation of the IRQL 0xFF situation. You can find this description in the Hector's Memos section of OSR Online.

So the answer to the question is that the first instruction in MmGetSystemRoutineAddress is paged out, plus interrupts are turned off. In another message to the mailing list, Tony traced through the OS code and found that, indeed, when interrupts are turned off that IRQL is set to 0xFF.

Tony said it with so much understatement: Now you just need to figure out why interrupts are disabled.

Indeed.

bool
bInterruptsAreDisabled()
{
        unsigned myEflags;

        __asm pushfd
        __asm pop             eax
        __asm mov             myEflags, eax

        bool bRet =
             (myEflags & 0x200) ? false : true;

        return bRet;
} /* bInterruptsAreDisabled() */

void
CrashIfInterruptsAreDisabled()
{
        if(bInterruptsAreDisabled())
        {
             KeBugCheck(?);
        }
} /* CrashIfInterruptsAreDisabled() */

OSR's Peter Viscarola wrote to me on the mailing list:

With the CrashIfInterruptsAreDisabled() code in place I was able to sprinkle defensive tests throughout my code.

At the beginning of driver entry I had:

#if DBG // {
            CrashIfInterruptsAreDisabled();

            MyBreakPoint();

            CrashIfInterruptsAreDisabled();
#endif // }

Where MyBreakPoint() is defined as:

// Call this if there is a constructor in the function. Will
// avoid an SEH/constructor error if using C++ in driver code
// (error C4509) that needs a destructor.
void
MyBreakPoint()
{
            __try { DbgBreakPoint() }
            __except(EXCEPTION_CONTINUE_EXECUTION) {}
}

I looked at the static crash dump. I saw that the second call to CrashIfInterruptsAreDisabled(); actually crashed my Toshiba laptop.

Oh dear god! It crashed on XP Professional, too! I sent the code to Jamey Kirby and he tested it and saw the same result.

Here's the enormous surprise: The following code turns off interrupts if there is no debugger running and does not turn it off if there is a debugger running!

Jamey points out that the Intel documentation states that INT 1, INT 3 and BOUNDS exceptions disable interrupts.

The above--seemingly trivial--code segment (which I and many, many other developers have been using for several years without a lick of trouble) was the source of the problem. It was the first statement in my device driver and it caused havoc millions of instructions later. Even more surprisingly, no one else has analyzed this problem and sounded the alarm bells in the developer community.

Jamey Kirby wonders how many seemingly spurious BSODs are the result of this " innocuous" piece of code.

The entire point of the above code is to break into a debugger if one is there but to continue execution if there is no debugger. This is the ultimate Heisenbug because the mere act of attempting to invoke the debugger causes a really nasty change in the state of the machine, from interrupts enabled to interrupts disabled.

We're now left with a another problem: How can one write code so that a debugger is safely tripped when a debugger is present but be harmless when there isn't a debugger present?

The first solution--a very bad solution--is to write assembly code to fix the problem.

if (KD_DEBUGGER_NOT_PRESENT == FALSE)
{
    // A kernel debugger is active.
    DbgPrint("A problem occurred\n");
    DbgBreakPoint();
}

Ignoring for the moment the fact that it doesn't work, why is this only a partial solution? Because according to the DDK documentation, "this global variable can only be used in Microsoft Windows XP and later." Because "If a kernel debugger was recently attached or removed, the value of KD_DEBUGGER_NOT_ PRESENT may not reflect the new state."

As to whether this solution works correctly: If you paste the code above into your driver and start up Windows without a kernel debugger running, you will get a BSOD. Bug check code 0x7E with parameter 1 being 0x80000003.

You guessed it--a breakpoint wasn't handled. Seems like we've been here before, right? Obviously, the sample code shown in the DDK documentation doesn't work!

Behold the Third Way!

Two caveats: This will only work on Windows XP and later systems. There is also the problem of a debugger being attached and then later detached. For that you'll need KdRefreshDebuggerNotPresent(). Alas, KdRefreshDebugger NotPresent() is only available for starting on Windows Server 2003.

Epilogue
After working for two years on this device driver--and after five weeks of hellish work so that we resolved what I hope is the last unresolved issue--Jamey told me that he thinks he has a different and better way to accomplish what we need... without needing a device driver.

I just love this profession.