It's Easy to be Hard -- Testing with HCTs
The NT Insider, Vol 11, Issue 3&4, May-August 2004 | Published: 18-Aug-04| Modified: 18-Aug-04
The Hardware Compatibility Tests, commonly called the HCTs, are a set of tests, documentation and processes that hardware/software is required to pass for participation in the "Designed for Microsoft Windows Logo Program" (http://www.microsoft.com/whdc/winlogo/default.mspx). But, they are actually more than that. They are a set of tests that many driver writers can use to directly or indirectly test their drivers – even if they do not directly control hardware and even if they are not going to submit their driver/device for certification.
This article explains what the HCTs are and how to use them.
What Are They?The HCTs are provided to developers so that they can pretest/preview the requirements that Microsoft has for hardware/software in order to ensure that quality standards necessary to obtain a Designed for Microsoft Windows logo are being met.
The Current HCTs support the testing of the hardware types seen in Figure 1.
- Audio (Adapter, Global Effects Filter, Synthesizer)
- Bus Controllers (Bluetooth,Cardbus/PCMCIA,1394,IrDA,and USB Controllers or Hubs)
- Display (Monitors)
- Anti-Virus/File System Filter Drivers
- Imaging (Cameras, Printers, Scanners)
- Input and HID (Game Devices, Keyboards, KVM Switches, Pointing and Drawing Devices, Smart Card Readers)
- Modems (Analog, Wireless, ISDN)
- Network Devices (ATM, Cable Modems, DSL, LAN, NDIS IM, NDIS Universal, Winsock Direct, WAN Devices)
- Storage Controllers and Devices (ATA/ATAPI,ATA RAID, Fibre, iScsi, Raid, SCSI, SATA, SATA RAID, CD-ROM, CD-RW, DVD, DVD-RW, DVD Combo, Tape, Hard Disks for all controllers , Removal Media, Cluster, RAID, Media Changer, Disk Storage Systems, Bridge Devices
- Streaming Media and Broadcast (DVD Playback, DirectX VA DVD Playback, Video Input and Capture)
- Systems (Desktop, Mobile, Motherboard, Server, Data Center Server, Cluster, Fault Tolerant, HAL)
- Unclassified (UPS, Miscellaneous Device, Universal Server Device)
Figure 1 -- HCT-Supported Hardware Device Types
The types of tests to be run and the testing configuration to be used vary depending on the category of the hardware/software. The HCT documentation outlines each category and the set of tests to be run. Testing for categories may contain both automatic and manual tests. For example, if you want to test either a keyboard device or a piece of software that is in the keyboard stack, you would use the HCTs for Keyboards as shown in Figure 2 (and as documented in Microsoft HCT documentation).
- ACPI Stress Test – Examines devices in the system, and then puts the system into a sleep/hibernation state. When the system returns from sleep/hibernation, the test verifies that the devices function properly after sleep/hibernation.
- ChkINF – Uses a Perl script that checks the structure and syntax of the device drivers INF files.
- Device Path Exerciser – Designed to crash a driver by calling in through various user-mode I/O interfaces. It tests driver robustness rather than driver functionality
- Driver Verifier – stresses device/driver combinations and tests the device’s use of system resources, including the driver’s memory paging behavior
- Public Import – Verifies that no APIs called by drivers are off-limits.
- SysParse – Inventories the devices, drivers, and software on the test system and records it. This test always passes, the information it gathers is used as part of the Logo submission process
- Winkey Media - Keyboard specific tests
- Winkey One-Key Combination – Keyboard specific tests
- Winkey Two-Key Combination – Keyboard specific tests
- Winkey Three-Key Combination – Keyboard specific tests
- IEEE 1394 - Run only the IEEE 1394 tests that are listed in Test Manager for your test system or device because your system or device may not require all 1394 tests.
- Cardbus/PCMCIA - Run only the Cardbus/PCMCIA tests that are listed in Test Manager for your test system or device because your system or device may not require all Cardbus/PCMCIA tests.
- PCI - Run only the PCI tests that are listed in Test Manager for your test system or device because your system or device may not require all PCI tests.
- USB- Run only the USB tests that are listed in Test Manager for your test system or device because your system or device may not require all USB tests.
Figure 2 -- Example: Keyboard-Specific HCT Tests
You might recognize a number of the tests above from the DDK where they’re also available. In most cases, when a tool appears both in the DDK and the HCT, the tools are the same (though this is not always true). And even when a tool does appear in both the DDK and the HCT, when the tool is run in the HCT for logo testing, it’s run with a particular set of options selected. These options may not be the ones you want to use, if you’re using the HCTs to exercise your driver’s functionality (and not to submit your driver for the Designed for Windows logo).
The HCT documentation describes each test in detail, including the approximate run time of the test, the output log file name, the OS that the test is supported on, test assertions, additional hardware requirements, and related links. So if you have a question on a test, all the documentation is there to help you find your answer.
Note: Though you may have no desire to "logo" your product, use caution in shipping a driver that has not passed the HCTs required for your specific device – your customer will take notice of system instability caused by your driver, and they won’t be happy about it.
Oh yeah…if you have a device/software that you want to "logo" and it does not appear in the provided device list, you can contact Microsoft and work with them to make it happen.
How Are They Used?When the HCT kit is installed (make sure that you select all the tests you want installed; not all tests are installed by default) it adds the HCT Test Manager, as seen in Figure 3, to your Start Menu. The Test Manager is the GUI application that allows the tester to select test categories, select tests, run tests, use test groups, view tests logs, monitor tests runs, and in general, manage the test process. It will automatically scan your test system for the test categories you installed and allow you to select, run and review the results of the relevant tests.
Figure 3 -- HCT Test Manager
For some categories of tests, it is possible to run tests from the command line. For example, the file system test, IFSTEST, the storage device stress utility, SdStress, and the NDIS test, NDTest, can be run from the command line
Another command line test worthy of mention is "devpathexer.exe" (fraternal twin to DC2 in the DDK). This nasty little program exercises your drivers IRP_MJ_DEVICE_CONTROL entry point. Many driver developers who support this entry point fail to perform all the necessary checks to ensure that each request that they receive is entirely valid, especially if they are supporting METHOD_NEITHER. Be it zero length buffers or invalid buffer addresses, most developers miss something, and fail to do the full set of rigorous tests necessary to truly shake-out their driver. DevPathExer will test this part of your code by sending it a variety of malformed requests. Passing this test will give you that extra level of confidence in your driver – before you’re made a fool of by your customer.
For more information on DC2, see the article On the Right Path.
The possible results for any one test are:
- A device required for this test is missing. The test cannot be run.
- The test has not been run successfully
- The test has been run but has not successfully passed.
- The test has been run and has successfully passed.
After completing the tests, the "Test Log Information" tab in the Test Manager allows the user to select logs for viewing. Figure 4 shows the log file for the "Disk Stress" test run on our test system.
Figure 4 -- Disk Stress HCT Log File
Not Just For Logo TestingOf course, the HCTs are useful for testing beyond fulfilling the Windows logo requirements. They can also be useful for doing general functionality testing for a driver.
Here at OSR, we do a lot of file system development. Thus, we use the HCT File System Test Specifications to verify the correctness of our file systems and file system filter drivers. As you probably know, file systems and file system filter drivers are some of the most difficult drivers to get right in Windows, so having a test suite is invaluable. The File System Test Descriptions provide a description of test "suites", the test "groups" that are included in the suite, and a description of the "tests" in each test group. So if you looked up "TD-21.15 ReadWrite File System Test Group" in Chapter 21 of the Anti-virus/File System Filter Driver Test Specification in the HCT Documentation, it would list the tests in this group and give a description of each of the tests.
The descriptions in the documentation are quite detailed, actually. In fact, it’s detailed enough that if you had to, you could write a test program to perform these tests yourself, if your file system was having problems in these areas. But why bother? The HCTs give you the ability to run individual tests to help you isolate problems. Why re-invent the wheel?
Things to RememberDon’t forget, it is also a good idea to write your own tests if your driver has additional functionality not tested by the HCTs. Further, remember that running and passing the HCTs
does not remove you from your professional obligation to utilize other standard debugging and testing practices such as running with Driver Verifier, using the checked build, CUV, etc.
SummaryThe HCTs are not only a requirement for the "Designed for Microsoft Windows" logo. They’re also a pretty good toolset for either directly or indirectly testing your driver. Whether running the tests via the HCT Test Manager or via the command line, the HCTs offers developers a well-documented set of tests that can help ensure that you are developing the highest quality driver possible.
The latest HCT downloads can be found by following links from http://www.microsoft.com/whdc/hwtest.
Rate this article and give us feedback. Do you find anything missing? Share your opinion with the community!
Post Your Comment