What’s the goal of a outfitting a test lab with test machines? Well, if you spend some time thinking about it, you’ll realize there are at least two separate goals that are dictated by the types of testing, you want to accomplish:

• Goal #1: You want to create environments that closely mirror those of your customers. In this case, you want to be able to create a facility that will allow you to reproduce customer scenarios (including the ability to repro problems that are reported in the field).
• Goal #2: You want to create environments that will allow you to achieve the broadest possible test coverage of your drivers. Here, your goal is to be able to perform general Q/A tests, including stress and regression.

One mistake that is often made in addressing goal #1 is believing that it’s more important, or even sufficient, to have systems that replicate the customer’s actual environment. While the ability to replicate customer environments is important, you have to remember that (for most test labs) you’ll never be able to replicate every possible customer environment. Also, don’t forget that customer environments will usually change and evolve. When this is the case, it’s usually very hard to keep a test lab outfitted with the latest and greatest systems that your customers might use.

That’s why it’s typically much more important to consider goal #2, and outfit a test lab with machines that’ll help you exercise and stress your code in the broadest possible ways. For example, at OSR we usually rotate machines from development use into our test lab. This helps keep the developers happy, and provides test machines that are at least slightly under-configured relative to the state of the art. We sometimes even remove memory from these machines. The goal for these machines is to stress our code in ways that it wouldn’t normally be stressed on a developer’s test system.

Putting your driver under heavy stress on a system that has a slow CPU and/or limited memory configuration, while also running other background applications, is very effective at forcing your driver to page, stress pool usage, and in general change the timing of your code execution drastically. How far do we take this? Well, would you believe that we have a dual-processor 90Mhz Pentium with 128MB of memory that we use for testing? We do. Yes, it’s annoying to use. But, we catch a lot of errors on this system when we crank-up IOMETER or our internal ActGen tool and start pounding!

On the other hand, you can’t just throw your cast-off junk into the test lab and claim you’re doing the right thing. If you’re going to find those tough timing problems, you need systems of many different speeds and capabilities, and that includes the high-end. It should go without saying that, multi-processor testing is a must. The more CPUs that you can afford to equip your test systems with, the better. A quad-processor system will be much more efficient at finding locking problems than a dual-processor system. The only large memory quad-processor system we’ve ever had at OSR lived in our test lab. These days, we’re waiting for our friends at Unisys to send us one of those awesome 32-processor systems that they make. Not that they ever said they would send us one, mind you. But, we can always keep hoping that one will spontaneously appear, right?

Another thing to consider is 64-bit support. Even if you’re not yet convinced that 64-bit support is important in your market (trust me, you will be convinced eventually) when you buy new test systems, why not get systems that can at least potentially perform double-duty? Here at OSR, when we recently expanded our test lab we added a dozen AMD64 systems. These systems offer the fastest 32-bit Windows performance available (for testing current releases of NT V4, Win2K, XP, and Server 2003 Sp1) and let us validate and test our drivers on the pre-releases of Windows-64 just by rebooting. It’s like getting a two-for-one deal.