We have encountered crashes on shutdown with our NDIS 5.1 WDM miniport
driver. The crashes may occur on less than 1 in 500 reboots on MP systems
with the latest multi-core CPUs. The problem seems to be a synchronization
problem. A common server has 2 NICs, but servers can have up to 32 NICs
installed. On shutdown the driver’s MiniportShutdown routine is called for
each NIC instance. The DDK states:
>MiniportShutdown does nothing more than restore the NIC to its initial
state
>(before the miniport’s DriverEntry function runs). However, this ensures
that the
>NIC is in a known state and ready to be reinitialized when the machine is
>rebooted after a system shutdown occurs for any reason, including a crash
>dump.
MiniportShutdown should not release any allocated resources.
>MiniportShutdown should just stop the NIC.
That is what we do. At the start of MiniportShutdown we set a flag in a
global driver structure that the NIC is shutting down. The NIC is stopped.
The receive DPC for the adapter will not pass up any more packets and any
queued send packets are returned by calling NdisMSendComplete. The driver may
queue send packets if it could not send the packets in the context of its
MiniportSendPackets routine. That could happen under heavy network load for
multiple NICs. But, in the case of the shutdown crashes the systems were
basically idle.
What we see is that the MiniportSendPackets routine is called by NDIS after
the MiniportShutdown routine has completed for the same adapter.
The crash occurs in the MiniportSendPackets routine, because the
MiniportAdapterContext handle that is passed to MiniportSendPackets is not
valid anymore.
The call stack is:
8089a09c f7862394 NetMp!MpSendPackets+0x4d
8089a0c4 ba80cf47 NDIS!ndisMSendX+0x1e4
8089a0ec ba807df8 tcpip!ARPSendData+0x198
8089a114 ba80c632 tcpip!ARPTransmit+0x11a
8089a250 ba809357 tcpip!_IPTransmit+0x84f
8089a2f0 ba80954a tcpip!UDPSend+0x445
8089a314 ba8095b0 tcpip!TdiSendDatagram+0xd5
8089a34c ba80907a tcpip!UDPSendDatagram+0x4f
8089a368 8081dce5 tcpip!TCPDispatchInternalDeviceControl+0x10c
8089a37c ba7d3cf8 nt!IofCallDriver+0x45
8089a398 ba7d3b9b netbt!TdiSendDatagram+0x14e
8089a3dc ba7d3fad netbt!UdpSendDatagram+0x14c
8089a424 ba7e6411 netbt!UdpSendNSBcast+0x288
8089a460 ba7e6513 netbt!ReleaseNameOnNet+0xb8
8089a48c ba7de8a2 netbt!NameReleaseDone+0x58
8089a4a8 ba7e6273 netbt!InterlockedCallCompletion+0x3c
8089a4c4 ba7cf054 netbt!ReleaseCompletion+0x14e
8089a4e4 808316c0 netbt!TimerExpiry+0x5b
Checking the miniport adapter states from the crash dump:
kd> !miniport 8a0db130
Miniport 8a0db130 : BladeFrame Network Management Veth, v0.0
AdapterContext : 88319000
Flags : ac452008
BUS_MASTER, IGNORE_TOKEN_RING_ERRORS, DESERIALIZED
RESOURCES_AVAILABLE, SUPPORTS_MEDIA_SENSE,
DOES_NOT_DO_LOOPBACK
MEDIA_CONNECTED, PM_HALTING,
PnPFlags : 01c10000
RECEIVED_START, NDIS_WDM_DRIVER, SHUT_DOWN
SHUTTING_DOWN,
MiniportState : STATE_UNDEFINED
IfIndex : 0
Ndis5MiniportInNdis6Mode : 0
InternalResetCount : 0000
MiniportResetCount : 0000
References : 3
UserModeOpenReferences: 0
PnPDeviceState : PNP_DEVICE_STARTED
CurrentDevicePowerState : PowerDeviceD0
Miniport Open Block Queue:
87c88ea0: Protocol 8a0e6e48 = NDISUIO, ProtocolBindingContext
8a0ecf58, v5.0
87e5c8d0: Protocol 881a9e58 = TCPIP, ProtocolBindingContext 87cc6c38,
v4.0
kd> !miniport 886bf680
Miniport 886bf680 : BladeFrame Network Veth0, v0.0
AdapterContext : 88665000
Flags : ac452008
BUS_MASTER, IGNORE_TOKEN_RING_ERRORS, DESERIALIZED
RESOURCES_AVAILABLE, SUPPORTS_MEDIA_SENSE,
DOES_NOT_DO_LOOPBACK
MEDIA_CONNECTED, PM_HALTING,
PnPFlags : 01c10000
RECEIVED_START, NDIS_WDM_DRIVER, SHUT_DOWN
SHUTTING_DOWN,
MiniportState : STATE_UNDEFINED
IfIndex : 0
Ndis5MiniportInNdis6Mode : 0
InternalResetCount : 0000
MiniportResetCount : 0000
References : 3
UserModeOpenReferences: 0
PnPDeviceState : PNP_DEVICE_STARTED
CurrentDevicePowerState : PowerDeviceD0
Miniport Open Block Queue:
8a176de0: Protocol 8a0e6e48 = NDISUIO, ProtocolBindingContext
87b26a28, v5.0
87cb2610: Protocol 881a9e58 = TCPIP, ProtocolBindingContext 87cb28e8,
v4.0
I put flags in the driver and confirmed that both instances of the adapters
had completed the MiniportShutdown routine. The PnPFlags values confirm that
NDIS set the state of miniports to SHUT_DOWN.
Checking for outstanding requests if found:
For the miniport Miniport 8a0db130 : BladeFrame Network Management Veth, v0.0
kd> !mopen 87c88ea0
Miniport Open Block 87c88ea0
Protocol 8a0e6e48 = NDISUIO, ProtocolContext 8a0ecf58, v5.0
Miniport 8a0db130 = BladeFrame Network Management Veth, v0.0
MiniportAdapterContext: 88319000
Flags : 00000000
References : 1
kd> !mopen 87e5c8d0
Miniport Open Block 87e5c8d0
Protocol 881a9e58 = TCPIP, ProtocolContext 87cc6c38, v4.0
Miniport 8a0db130 = BladeFrame Network Management Veth, v0.0
MiniportAdapterContext: 88319000
Flags : 00000000
References : 1
For the miniport Miniport 886bf680 : BladeFrame Network Veth0, v0.0
kd> !mopen 8a176de0
Miniport Open Block 8a176de0
Protocol 8a0e6e48 = NDISUIO, ProtocolContext 87b26a28, v5.0
Miniport 886bf680 = BladeFrame Network Veth0, v0.0
MiniportAdapterContext: 88665000
Flags : 00000000
References : 1
kd> !mopen 87cb2610
Miniport Open Block 87cb2610
Protocol 881a9e58 = TCPIP, ProtocolContext 87cb28e8, v4.0
Miniport 886bf680 = BladeFrame Network Veth0, v0.0
MiniportAdapterContext: 88665000
Flags : 00000000
References : 2
The reference count of 2 for the TCPIP protocol makes sense to me, because
it has an outstanding packet to the miniport.
I confirmed that under normal operation. Whenever MiniportSendPackets is
called the TCPIP protocol reference count is 1+NumberOfPackets.
That leads me to believe that our driver does not have any outstanding
packets for the adapter when it returns from its MiniportShutdown routine. At
that point all send packets are returned to NDIS and all receive packets have
been returned from NDIS. (I confirmed that by checking the packet count of
the allocated and returned receive packets.)
Since MiniportSendPackets could be called or in process when a shutdown or
adapter halt occurs we need to account for that. In the case of a shutdown we
simply completes the packets. MpSendPacket also checks if the adapter can
send packets. If a packet could not be sent and was not queued (e.g. the
adapter is shutting down or is halted) then the packet is simply returned to
NDIS.
MiniportSendPackets does the following:
VOID MiniportSendPackets (IN NDIS_HANDLE MiniportAdapterContext,
IN PPNDIS_PACKET PacketArray,
IN UINT NumberOfPackets)
{
PMY_ADAPTER Adapter = (PMY_ADAPTER)MiniportAdapterContext;
UINT i;
NDIS_STATUS Status = NDIS_STATUS_SUCCESS;
PNDIS_PACKET pPacket;
if (pGlobal->Flags & ADAPTER_SHUTDOWN)
{
if ((Adapter && Adapter->MpHandle)
{
IncRefCount(Adapter);
for (i = 0; i < NumberOfPackets; i++)
{
pPacket = PacketArray[i];
NdisMSendComplete(Adapter->MpHandle,
pPacket, NDIS_STATUS_ADAPTER_REMOVED);
}
DecRefCount(Adapter);
}
return;
}
// send packets
IncRefCount(Adapter);
for (i = 0; i < NumberOfPackets; i++)
{
pPacket = PacketArray[i];
Status = MpSendPacket(Adapter, pPacket);
if (Status != NDIS_STATUS_PENDING)
{
NdisMSendComplete(Adapter->MpHandle
Packet, NDIS_STATUS_SUCCESS);
}
}
KeSetEvent (&pGlobal->WakePacketCompletionThreadEvent, IO_NO_INCREMENT,
FALSE);
DecRefCount(Adapter);
}
The problem is that in the case of a shutdown the MiniportAdapterContext
pointer may not be valid anymore. The system crashes on the reference to
Adapter->MpHandle.
I don’t understand why NDIS is calling the Miniport’s send routine after it
has shutdown the miniport. Should we in that case NOT call NdisMSendComplete,
assuming that the system is shutting down anyway? Or do we have a bug in our
driver caused by an outstanding packet or handle reference that is not
reflected by the counters available through ndiskd?
I went through the exercise and cleaned up all the miniport resources on
shutdown (in contradiction to the DDK), but I still see the same problem:
The MiniportSendPackets routine is called by NDIS after the MiniportShutdown
routine has completed for the same adapter. And it does not matter if we have
one or multiple adapters in the system. It may just take longer to reproduce
the problem with just one adapter.
Gernot Seidler
Windows Engineering
http://www.egenera.com