Driver Problems? Questions? Issues?
Put OSR's experience to work for you! Contact us for assistance with:
  • Creating the right design for your requirements
  • Reviewing your existing driver code
  • Analyzing driver reliability/performance issues
  • Custom training mixed with consulting and focused directly on your specific areas of interest/concern.
Check us out. OSR, the Windows driver experts.

OSR Seminars


Go Back   OSR Online Lists > ntdev
Welcome, Guest
You must login to post to this list
  Message 1 of 8  
27 Dec 17 02:27
Michael Rolle
xxxxxx@rolle.name
Join Date: 27 Dec 2017
Posts To This List: 59
How to find an available interrupt vector that doesn't conflict.

I've written a driver to access the Instruction-Based Sampling (or IBS) hardware on newer AMD CPUs. I need to select an interrupt vector, which I can link to my ISR and also tell the IBS hdwe which vector to generate its interrupt on. The hdwe doesn't have a predefined vector, although it does have a predefined LVT entry in the local APIC. Being a newbie to windows drivers, and not wanting to take the time to master the art completely, I came up with a method. I have a four-level loop where I try IoConnectInterrupt with various parameter values. In order, the loop tries: * exclusive, shared access * latched, level sensitive * vectors 50 through FE * IRQLs 0 through 9. On my system, the first success is with exclusive, latched, vect 80h, IRQL 0. I've also tried testing shared access first, then the result is shared, with the other values the same. Unfortunately, after enabling IBS interrupts and servicing several of them, the system completely hangs. No mouse, no keyboard, no console update for my client that is monitoring progress. This code used to work a few years ago, when I had Windows 7 and an earlier AMD processor. Now I've got Windows 10 and an AMD Ryzen and I get the hangups. I figure I was just plain lucky that it used to work before. Anyway, I'm sure there's a RIGHT way to do this, but I don't know what that would be. I'm asking for help from you all, at least pointers to stuff I can read. I've got a few inklings... 1. I need to use the PnP manager. Even though the IBS hdwe is not an external device, I have the impression that PnP makes sure everything works in cooperation. 2. Maybe the DIRQL of 0 is wrong, and should be higher? 3. There's something in the Hal layer that manages interrupt vectors, which I can use. 4. Reserving an interrupt vector for the IBS hdwe in the Registry somehow. Another bit of info: AMD has a free program called CodeXL that takes IBS samples. Its kernel drivers operate without system hangups, so obviously it's possible. I'm going through disassembly of the driver code to figure out what it's doing to get its ISR called. So far, I've seen a call to HalAllocateHardwareCounters, which will prevent other (well-behaved) drivers from accessing the IBS hardware. Perhaps this is what I need to do in my own IBS driver too? Here is the list of relevant imports for the two CodeXL drivers, if that may help you discern what their method is. Curious that IoConnectInterrupt(Ex) is not among them. Do you recognize which of these functions will bind an ISR to an interrups? DLL Name: ntoskrnl.exe ExFreePoolWithTag KeInitializeEvent KeWaitForSingleObject DbgPrint IoAllocateWorkItem ExRegisterCallback ObfDereferenceObject ExCreateCallback RtlInitUnicodeString IoQueueWorkItemEx IoFreeWorkItem RtlIsNtDdiVersionAvailable HalDispatchTable ExUnregisterCallback KeGetCurrentProcessorNumberEx IoDeleteDevice IoDeleteSymbolicLink IoCreateDevice ExAllocatePoolWithTag IoCreateSymbolicLink KeQueryActiveProcessorCountEx PsGetCurrentProcessId PsGetCurrentThreadId MmIsAddressValid KeAcquireSpinLockRaiseToDpc KeRemoveQueueDpc ExSetTimerResolution KeSetTimerEx KeInitializeTimer KeReleaseSpinLock KeCancelTimer KeClearEvent KeSetEvent KeInitializeDpc KeInsertQueueDpc KeGetProcessorNumberFromIndex KeSetTargetProcessorDpcEx MmMapIoSpace MmUnmapIoSpace IofCompleteRequest KeSetImportanceDpc __C_specific_handler DLL Name: HAL.dll KeQueryPerformanceCounter HalAllocateHardwareCounters HalFreeHardwareCounters HalGetBusDataByOffset DLL Name: ntoskrnl.exe ObReferenceObjectByHandle ExAllocatePoolWithTag KeReleaseGuardedMutex ExFreePoolWithTag KeAcquireGuardedMutex ExEventObjectType DbgPrint KeGetCurrentProcessorNumberEx MmUnlockPages PsRemoveLoadImageNotifyRoutine ZwOpenSection ZwUnmapViewOfSection MmProbeAndLockPages PsSetLoadImageNotifyRoutine IoAllocateMdl RtlInitUnicodeString PsRemoveCreateThreadNotifyRoutine ZwMapViewOfSection IoFreeMdl MmMapLockedPagesSpecifyCache PsSetCreateThreadNotifyRoutine PsSetCreateProcessNotifyRoutine ZwClose IoDeleteDevice IoDeleteSymbolicLink IoCreateDevice IofCompleteRequest IoCreateSymbolicLink KeQueryActiveProcessorCountEx ZwCreateFile ZwWriteFile ZwQueryInformationFile KeSetEvent ExInterlockedInsertTailList ExInterlockedRemoveHeadList PsIsThreadTerminating PsCreateSystemThread KeClearEvent KeWaitForSingleObject PsTerminateSystemThread KeSetImportanceDpc KeInsertQueueDpc KeInitializeDpc RtlQueryRegistryValues KeInitializeEvent IoSizeofWorkItem ZwQueryVolumeInformationFile IoQueryFileDosDeviceName IoInitializeWorkItem IoQueueWorkItemEx ObfReferenceObject IoUninitializeWorkItem ZwOpenFile IoIs32bitProcess ObfDereferenceObject KeInitializeGuardedMutex IoGetStackLimits RtlVirtualUnwind MmSystemRangeStart RtlQueryModuleInformation MmHighestUserAddress ProbeForRead ExpInterlockedFlushSList KeInitializeApc ExpInterlockedPopEntrySList ExpInterlockedPushEntrySList PsGetCurrentProcessId KeInsertQueueApc PsGetThreadId PsGetThreadProcessId InitializeSListHead RtlImageDirectoryEntryToData RtlIsNtDdiVersionAvailable KeGetProcessorNumberFromIndex KeSetTargetProcessorDpcEx PsGetCurrentProcessWow64Process MmIsAddressValid __C_specific_handler DLL Name: HAL.dll HalRequestSoftwareInterrupt KeQueryPerformanceCounter
  Message 2 of 8  
27 Dec 17 14:05
anton bassov
xxxxxx@hotmail.com
Join Date: 16 Jul 2006
Posts To This List: 4487
How to find an available interrupt vector that doesn't conflict.

<quote> Being a newbie to windows drivers, and not wanting to take the time to master the art completely, I came up with a method. I have a four-level loop where I try IoConnectInterrupt with various parameter values. In order, the loop tries: * exclusive, shared access * latched, level sensitive * vectors 50 through FE * IRQLs 0 through 9. </quote> Why haven't you tried hitting your test computer with a hammer(or at least writing zero to CR0) just to check if it may be of any help? Look - in terms of chances of success your "methodology" does not seem to fare any better than the one that I have suggested...... BTW, a combination of IRQL 0 with interrupt vector 0xFE seems to be particularly impressive ..... Anton Bassov
  Message 3 of 8  
27 Dec 17 15:00
Daniel Terhell
xxxxxx@resplendence.com
Join Date: 15 Apr 2004
Posts To This List: 926
How to find an available interrupt vector that doesn't conflict.

Taking a quick look, this thing doesn't appear to be interrupt driven but like most CPU features uses MSRs. Why don't you start out with the Linux sample that's easy to find ? https://github.com/jlgreathouse/AMD_IBS_Toolkit/blob/master/README.txt //Daniel
  Message 4 of 8  
27 Dec 17 17:55
Tim Roberts
xxxxxx@probo.com
Join Date: 28 Jan 2005
Posts To This List: 11957
How to find an available interrupt vector that doesn't conflict.

xxxxx@rolle.name wrote: > I've written a driver to access the Instruction-Based Sampling (or IBS) hardware on newer AMD CPUs. > ... > This code used to work a few years ago, when I had Windows 7 and an earlier AMD processor. Now I've got Windows 10 and an AMD Ryzen and I get the hangups. I figure I was just plain lucky that it used to work before. > > Anyway, I'm sure there's a RIGHT way to do this, but I don't know what that would be. The RIGHT way to do this is to have an entry in the BIOS ACPI DSDT for this interrupt.  Then, Windows will create an interrupt with the parameters already established, and will pass the resource to whatever driver gets loaded for that ACPI device. Why does this process need an interrupt? > 2. Maybe the DIRQL of 0 is wrong, and should be higher? Absolutely, it is wrong.  IRQL of 0 is PASSIVE_LEVEL  DIRQLs are typically equal to the vector number (although that could have been an architectural accident on the machines where I was looking). > Here is the list of relevant imports for the two CodeXL drivers, if that may help you discern what their method is. Curious that IoConnectInterrupt(Ex) is not among them. Do you recognize which of these functions will bind an ISR to an interrups? Does that driver have an INF file?  Is it matching an ACPI device code?  Or is it a legacy driver that's attaching to the interrupt through hackery?  The presence of HalRequestSoftwareInterrupt suggests hackery, although AMD probably participated in writing the HAL and understands its proper use. -- Tim Roberts, xxxxx@probo.com Providenza & Boekelheide, Inc.
  Message 5 of 8  
28 Dec 17 00:45
Michael Rolle
xxxxxx@rolle.name
Join Date: 27 Dec 2017
Posts To This List: 59
How to find an available interrupt vector that doesn't conflict.

Daniel, thanks and I'll check out the link you gave me. But if it's for Linux, it won't be much help, as I already know how to use the MSRs for the IBS. Tim, I don't know how CodeXL's drivers are installed. When I install CodeXL, the drivers appear in System32/drivers and in the SCM's database. I suspect that the call to HalRequestSoftwareInterrupt is not what registers the ISR. I couldn't find any documentation for this routine. From the disassembly, it appears that it is being called with a single argument of 1 in the CL register. Anyway, this call is in one of the drivers, and the actual ISR, as far as I can tell, is in the other driver and it's address is not exported. As I mentioned before, do you recognize any functions in the FIRST group that will hook an interrupt vector? I did find in the AMD driver where it sets the vector number in the LVT entry. The number is 1. The rest of the bits in the LVT are set to 0. Does that make sense to you, that interrupt 1 would be dedicated to the IBS? I'm still looking to find the place where the vector is hooked to the ISR. AMD's IBS uses interrupts because it has a cycling counter, and when the counter rolls over, it signals an interrupt (if they are enabled). The ISR can then read information from various MSRs and reenable interrupts and sampling for the next counter rollover. This way, the driver can capture periodic samples of whatever is running on the core where the ISR is hooked to the interrupt. Now I could have my driver just read the MSRs on a control request from the client, and return the sample values if they exist, and reenable sampling for the next rollover. All this without using interrupts. If I can't get interrupts to work right, I'll have to resort to this and have my client app use the driver to poll the IBS. It won't be as good for my purposes, but it will do if need be. Regarding DIRQL = 0... I wasn't sure if IoConnectInterrupt would succeed for some values and not for others, so I decided to try a whole range. Now maybe all I need to do is pick a level and just use that. Is DISPATCH_LEVEL the lowest level that is valid for an ISR? Or does it need to be higher? Regarding the interrupt vector... If AMD and the BIOS are doing the "right" thing, it may be that there's something in the BIOS that determines the interrupt vector. I know that the hardware uses a dedicated APIC LVT entry, whose index is given in an MSR. I assumed I had to choose a vector and then store it in the LVT entry. But perhaps the BIOS and/or the system has selected a vector and stored it there already. I'll look at the APIC LVT initial state before I try to change it and see if there's something meaningful there. If not, can you tell me how to explore the BIOS ACPI DSDT from my app, or from a kernel driver, to see if there's anything there regarding the IBS. Can you give me a reference explaining what this DSDT is and its format?
  Message 6 of 8  
28 Dec 17 02:28
anton bassov
xxxxxx@hotmail.com
Join Date: 16 Jul 2006
Posts To This List: 4487
How to find an available interrupt vector that doesn't conflict.

> I suspect that the call to HalRequestSoftwareInterrupt is not what registers the ISR. Indeed..... > I couldn't find any documentation for this routine. This routine requests an interrupt on the CPU (either a self-interrupt or IPI, depending in the argument) by writing to the local APIC's ICR. It is not supposed to be used by driver writers,and this is the reason why it is not documented..... > Regarding DIRQL = 0... I wasn't sure if IoConnectInterrupt would succeed for some values > and not for others, so I decided to try a whole range. Now maybe all I need to do is > pick a level and just use that. What yo really need to do is to learn what the very concept of IRQL is all about, and then discover how it is actually implemented. At this point you will realize how stupid all your"experimentation" is. Basically, this is just an index into the array that stores vector numbers - the higher the index is, the higher the numerical value of the vector is. Interrupt priority is implied by the vector number on x86 and x86_64 systems > do you recognize any functions in the FIRST group that will hook an interrupt vector? The only functions that hook an ISR are IoConnectInterrupt()and IoConnectInterruptEx(). However, your are supposed to call these functions only with the right parameters, i.e. the ones that you get from resource descriptors. What they do is registering your ISR with a call chain that gets invoked by interrupt handler stub (i.e. the one that the address specified in IDT points to) when an interrupt that corresponds to a given vector number occurs. It has absolutely nothing to do with mapping IRQ lines to vectors or writing a handler address to IDT. This part gets done at the boot time, based upon the information that the OS gets from BIOS (i.e. concerning the relationship between the PCI/ PCI-X devices and interrupt lines and pins) . In order to realize how stupid your "experimentation"is, consider the scenario when you physically share an interrupt line and pin with another PCI device(s), which is specified in the BIOS settings, but still want to get a separate interrupt vector and provide your very own arguments to IoConnectInterrupt().... Anton Bassov
  Message 7 of 8  
28 Dec 17 13:28
Tim Roberts
xxxxxx@probo.com
Join Date: 28 Jan 2005
Posts To This List: 11957
How to find an available interrupt vector that doesn't conflict.

xxxxx@rolle.name wrote: > Tim, I don't know how CodeXL's drivers are installed. When I install CodeXL, the drivers appear in System32/drivers and in the SCM's database. > ... > Regarding the interrupt vector... If AMD and the BIOS are doing the "right" thing, it may be that there's something in the BIOS that determines the interrupt vector. I know that the hardware uses a dedicated APIC LVT entry, whose index is given in an MSR. I assumed I had to choose a vector and then store it in the LVT entry. But perhaps the BIOS and/or the system has selected a vector and stored it there already. Perhaps.  Have you actually asked AMD about this?  I assume they have the same kind of detailed support forums that Intel has, and those forums are usually pretty active.  I'm sure there are AMD nuts on their forums who would be happy to tell you way more than you want to know about this. -- Tim Roberts, xxxxx@probo.com Providenza & Boekelheide, Inc.
  Message 8 of 8  
29 Dec 17 20:04
Michael Rolle
xxxxxx@rolle.name
Join Date: 27 Dec 2017
Posts To This List: 59
How to find an available interrupt vector that doesn't conflict.

Thanks, Tim and Anton. The short story is that, apparently, the CodeXL driver does not use interrupts from the IBS hardware at all, hence no interrupt vector is involved. Rather, it uses a timer interrupt to periodically poll the IBS to see if a sample is available. Regarding help from AMD community, there is a DevGurus forum, but there's very little activity on it. They have a CodeXL subforum which hasn't had any activity since 4 months ago. The general discussions subforum is more active. I'm going to join the forum and post my question and see. Also, there's a general email for AMD support that I'll try. If I learn anything I'll post it here. I was mistaken about using vector number 1. I thought that the value was being stored in a word that would later go into the LVT, but I had the wrong base register in the disassembly. So I still don't know where the driver is getting the vector number. Since there is no call to IoConnectInterrupt(Ex), I am going to assume the driver is using a repeating timer interrupt. In fact, the function that checks the IBS hardware for a rollover and reads the info MSRs and reenables the sampling is called Deferred_TimerInterrupt Service, and looks like it is scheduled by KeInitializeTimer. I'm guessing also that where the driver sets the IBS' LVT entry, it is doing so to disable the interrupt, rather than enable it and store a vector. The IBS hardware always signals an interrupt when a sample is taken, and the LVT determines whether the processor is actually interrupted.
Posting Rules  
You may not post new threads
You may not post replies
You may not post attachments
You must login to OSR Online AND be a member of the ntdev list to be able to post.

All times are GMT -5. The time now is 21:36.


Copyright ©2015, OSR Open Systems Resources, Inc.
Based on vBulletin Copyright ©2000 - 2005, Jelsoft Enterprises Ltd.
Modified under license