The NT Insider:Sometimes You Have to Write Your Own -- Case Study: ActGen IO Utility

Everything Windows Driver Development

Thu, 14 Mar 2019 118020 members

Online Dump Analyzer
OSR Dev Blog
The NT Insider
The Basics
File Systems
Downloads
ListServer / Forum


	Express Links

	·	The NT Insider Digital Edition - May-June 2016 Now Available!
	·	Windows 8.1 Update: VS Express Now Supported
	·	HCK Client install on Windows N versions
	·	There's a WDFSTRING?
	·	When CAN You Call WdfIoQueueP...ously

THE NT INSIDER

Sometimes You Have to Write Your Own -- Case Study: ActGen IO Utility
The NT Insider, Vol 11, Issue 3&4, May-August 2004 | Published: 18-Aug-04| Modified: 18-Aug-04

The ability to generate native Windows I/O API events could be an invaluable process in testing functionality in Windows drivers. For file system drivers in particular, this level of testing is absolutely necessary. Alas, it isn’t possible to test all variations of the native Windows API using Win32 calls, since Win32 is implemented as a subsystem. That is, you’re really talking to an emulation layer when you use the Win32 API, which precludes you from testing things such as extended attributes, case sensitivity, open by file id, and the enumeration of streams.

So where does that leave us? Well, we do know that the Win32 API, used by applications and services, is translated into native system calls – the infamous "native" Windows API. However, many of the native function calls beginning with NtXxx are undocumented and unsupported by Microsoft, which makes working with the native Windows API complicated. Also (and more to the point) who wants to write a program every time you wanna test a specific feature? A tool to simplify the process of generating native I/O system service requests would be most welcome.

A Potential Solution
The OSR I/O Activity Generator (ActGen) is a development support and test utility. It is one possible solution to the above stated need. ActGen facilitates the development of tests, provides an infrastructure to directly access the native Windows API and has proven to be an invaluable tool in testing Windows drivers here at OSR. The foundations of ActGen are prototypes for the native Windows NtXxx functions accessed via a Windows Console application and commanded with script-based tests. Hopefully, this introduction to ActGen will provide fodder for your own investigations in this area.

API Prototypes
Given the undocumented nature of much of the native Windows API, securing prototype functions to develop ActGen required a bit of effort. Trial and error, as well as careful observations of system behavior went a long way, and there certainly are more public sources available these days to provide insight (and in some cases, prototypes themselves) to generate a useful list to work with. In the end, we determined prototypes for the NtXxx functions as best we could. With native Windows API prototypes "in hand" we were ready to develop our script-based tool.

The Scripting Language
The scripting language we developed is very easy to understand and, hence, easy to use. ActGen is started from a Windows Console Prompt and processes script commands from a user-written script file, which is passed to ActGen upon invocation. Commands are processed from the test script until the end of the script file is reached or a script-terminating failure occurs. Using a lexical parser, ActGen reads an input line from the test script and calls the appropriate functions to get command-specific parameters and then actually performs processing for the command. Since it’s important that ActGen is able to generate invalid requests, the tool does not in any way attempt to validate the commands sent to it.

An Example Test Script
So what kinds of things can you do with ActGen?

Here is a simple example of how one would run the test script mytest.act and indicate that it should be run on drive F:

C:> actgen mytest.act –d F:

Let’s dissect a sample test script written to test files opened with read-only access. We’ll start by looking at the beginning of the script in Figure 1.

!----------------------------------------------------------------------------------------------------

! Before we begin, make sure the directory being used in this script exists.

! Create the directory if it does not exist.

!-----------------------------------------------------------------------------------------------------

CREATE 77, "$DISK1$\OSRTEST", GENERIC_ALL, DIRECTORY, FILE_OPEN_IF

CLOSE 77

!-------------------------------------------------------------------------

! Create a file, write some data to it, and close it

!-------------------------------------------------------------------------

Create 1 "$DISK1$\OSRTEST\TEST1.DAT",GENERIC_WRITE,CREATE_ALWAYS

MWrite 1 100, 512, 0

Close 1

Figure 1 -- Sample Script: Testing Create

The Create command can be used to create/open directories or files. To override the default of creating a file, use the flag DIRECTORY to specify that a directory is to be created. Notice that file numbers are specified when a file/directory is created/opened. This number is then used to identify which file a command is to be run against.

You’re probably wondering about the presence of $DISK1$. This is the string that will be replaced with the drive letter passed in to ActGen with the –d option. In the ActGen invocation example above, the driver letter F: is passed in so $DISK1$ will be replaced with F:. It is important to note that in addition to drive letters, UNC path specification can also be passed in the with the –d option. Also, if your script requires more than one drive letter or UNC path, you can supply up to 26 drive strings to replace $DISKxx$ values in the test script.

Create dispositions can be specified to indicate how to create/open a file. In the above example, FILE_OPEN_IF is used to ensure that the directory where data files will be stored already exists. If it doesn’t exit, then the directory will be created. For the creation of the data file, the disposition CREATE_ALWAYS is used to ensure that a new file will be created, even if a file with that name already exists.

Once we’ve ensured that the directory exists, and we’ve created a file in the test directory, we can write some data to the file to be used for this test. In this example we use Mwrite to perform multiple write requests. A total of 100 writes of 512 bytes each will be performed, and the first write will be at offset 0 – the beginning of the file.

Let’s continue to look at this script in Figure 2.

!-------------------------------------------------------------------------------------------------

! Now open the file as read-only and make sure that you can read but not

! write or delete.

!---------------------------------------------------------------------------------------------------

Create 1 "$DISK1$\OSRTEST\TEST1.DAT", GENERIC_READ, OPEN_EXISTING

MRead 1 100, 512, 0

ReadAsync 1 512,0

Expect 1 0xc0000022

Write 1 512,0

WriteAsync 1 512,512

MWrite 1 2, 512, 0

Delete 1

Expect 1 Success

Close 1

!---------------------------------------------------------------------------------------

! Check to be sure the file can be opened to ensure delete failed

!-----------------------------------------------------------------------------------------

Create 1 "$DISK1$\OSRTEST\TEST1.DAT", GENERIC_ALL, OPEN_EXISTING

Close 1

Figure 2 -- Multiple Read Requests (Mwrite)

This time the file is opened with only read access by specifying GENERIC_READ as a create access option. Notice that the file is opened with the create disposition of OPEN_EXISTING because we expect this file to exist since it was just created. If it does not exist, there’s a problem and we need to know about it.

At this point we want to ensure that reads succeed and writes fail. To do this, we test the different kinds of read requests to ensure they complete with STATUS_SUCCESS and test different writes to ensure they complete with STATUS_ACCESS_DENIED.

Like its counterpart, MRead performs multiple read requests. To reflect the write done when this file was created, 100 reads of 512 bytes each is performed, and the first read is at offset 0 – the beginning of the file. Both a synchronous read, Read, and also an asynchronous read, ReadAsync, are performed, each specifying that 512 bytes be read but starting at different offsest in the file.

By default, ActGen assumes that the expected command completion status will be STATUS_SUCCESS (0x0000000). However, if you know the command you are about to issue is going to fail, you can specify the expected completion status with the Expect command. For all the write tests and the delete to be performed on this file opened with read access, we expect STATUS_ACCESS_DENIED to be returned. Therefore, we tell ActGen that we expect a completion code of 0xC0000022. Once we have completed this set of tests we can set the expected status back to the default with Expect 1 Success.

As a way of double checking that the file did not get deleted upon close, due to the issuing of the Delete command, we open the file with the create disposition of OPEN_EXISTING to ensure that the file still exists. If the file has been deleted and does not exist, the script will fail because the expected status is different than the actual completion status.

Mapped Reads can also be tested with ActGen, see Figure 3.

!-------------------------------------------------------------------------

! Check MAPPED_READ access as well...

!-------------------------------------------------------------------------

Create 1 "$DISK1$\OSRTEST\TEST1.DAT", MAPPED_READ, OPEN_EXISTING

MRead 1 100, 512, 0

ReadAsync 1 512,0

Close 1

Figure 3 -- Testing Mapped Read Access

Memory mapped read access is tested by specifying MAPPED_READ as the desired access on open of the file. As in the previous code section, we test the various kinds of read requests to ensure they all succeed.

Cleaning up files used during the test is the last step in this script and can be seen in Figure 4.

!--------------------------------------------------------------------------

! Clean up files so this script will work next time

!---------------------------------------------------------------------------

Create 1 "$DISK1$\OSRTEST\TEST1.DAT", GENERIC_ALL, FILE_OPEN, DELETE_ON_CLOSE

Close 1

CREATE 77, "$DISK1$\OSRTEST", GENERIC_ALL, DIRECTORY, FILE_OPEN

DELETE 77

CLOSE 77

:end

!END OF SCRIPT

Figure 4 -- Clean Up of Files

Other Features of the Scripting Language
In addition to the commands described in the example, the ActGen scripting language is capable of testing many other native interfaces. ActGen can be used to get and set extended attributes, see Figure 5.

!--------------------------------------------------------------------------------------------------------

! Add three new EAs to the file. After this, there should be 4 EAs on the file.

!---------------------------------------------------------------------------------------------------------

SetEa 1 "FRED",300

SetEa 1 "DICK",200

SetEa 1 "HARRY",225

!---------------------------------------------------------------------------------

! List all EAs on the file, there should be 4 EAs on the file.

!---------------------------------------------------------------------------------

GetEa 1

!---------------------------------------------------------------------------------

! Now try to list DICK and HARRY

!---------------------------------------------------------------------------------

GetEa 1 "HARRY,DICK"

Figure 5 -- Get and Set Extended Attributes

Extended Attributes can be set on an open file using the SetEa command and then listed using the GetEa command. All extended attributes defined for the file are listed by default with the GetEa command. To list just certain extended attributes, simply specify the name of the extended attribute(s) to be listed as the second parameter of the GetEa command.

Script flow control statements allow for jumping to other areas of the script with the stand alone Goto command or based on file system name and the file system mount characteristics. File locking can also be tested with ActGen, see Figure 6.

!---------------------------------------------------------------------------------

! Open the file for READ

!---------------------------------------------------------------------------------

CREATE 1 "$DISK1$\OSRTEST\lock1.dat", GENERIC_READ, OPEN_EXISTING, SHARE_ALL

!---------------------------------------------------------------------------------

! Open the file a second time

!---------------------------------------------------------------------------------

CREATE 2 "$DISK1$\OSRTEST\lock1.dat", GENERIC_READWRITE, OPEN_EXISTING, SHARE_READ

!---------------------------------------------------------------------------------

! Now take a lock out on the first open instance of the file

!---------------------------------------------------------------------------------

! Length, Starting Offset

Lock 1, 200, 0

!--------------------------------------------------------------------------------------------------------------

! Now attempt to read the locked region, using the second open instance of the file

!-------------------------------------------------------------------------------------------------------------

Expect 2, 0xc0000054

Expect 2, Success

!---------------------------------------------------------------------------------

! Unlock and close the file

!---------------------------------------------------------------------------------

Unlock 1, 200, 0

CLOSE 1

CLOSE 2

Figure 6 -- File Locking

We open two instances of an existing file and lock a range of the file. We can then use the second open instance of the file to ensure that the locked portion of the file cannot be read and that unlocked portions of the file can be read.

Multi-Threaded Operation and Synchronization
One of ActGen’s more powerful features is the ability to spawn multiple threads and synchronize the operation of these spawned threads. Test scripts can be run synchronously or asynchronously and they can be grouped to allow for synchronization using SYNC points. Let us show you with an example.

We’ll use three scripts in this simple example: a master script (master.act), see Figure 7, an asynchronous script (async.act), see Figure 8 and a synchronous script (sync.act), see Figure 9.

!-------------------------------------------------------------------

! Define the start of a group and provide a name

!-------------------------------------------------------------------

GroupStart “SimpleGroup”

!---------------------------------------------------------------------------

! This script is run asynchronously in its own name space

!----------------------------------------------------------------------------

run “async.act”

!--------------------------------------------------------------------------

! This script is run synchronously within the name space

! of the master script

!---------------------------------------------------------------------------

@sync.act

!-------------------------------------------------------------------

! End of group

!-------------------------------------------------------------------

GroupEnd

!----------------------------------------------------------------------------

! End of script

!----------------------------------------------------------------------------

Figure 7 -- MASTER.ACT

!-------------------------------------------------------------------

! Create a file in an existing directory

!-------------------------------------------------------------------

CREATE 77, "C:\OSRTEST", GENERIC_ALL, DIRECTORY, FILE_OPEN_IF

CLOSE 77

Create 1 “C:\OSRTEST\TEST5.DAT", GENERIC_ALL, CREATE_ALWAYS

MWrite 1 100, 512, 0

Delete 1

Close 1

!----------------------------------------------------------------------------------------------

! Set a sync point and wait “forever” for sync wait conditions to be met

!----------------------------------------------------------------------------------------------

SYNC “CompletionPoint”,-1

!-------------------------------------------------------------------------------------------

! Once the sync point has been reached, we can delete the directory

!---------------------------------------------------------------------------------------------

CREATE 77, "C:\TEST", GENERIC_ALL, DIRECTORY, FILE_OPEN_IF

DELETE 77

CLOSE 77

!----------------------------------------------------------------------------

! End of script

!----------------------------------------------------------------------------

Figure 8 -- ASYNC.ACT

!-------------------------------------------------------------------

! Create a file in an existing directory

!-------------------------------------------------------------------

CREATE 77, "C:\OSRTEST", GENERIC_ALL, DIRECTORY, FILE_OPEN_IF

CLOSE 77

Create 1 “C:\OSRTEST\TEST6.DAT", GENERIC_ALL, CREATE_ALWAYS

MWrite 1 100, 512, 0

Delete 1

Close 1

!----------------------------------------------------------------------------------------------

! Set a sync point and wait “forever” for sync wait conditions to be met

!----------------------------------------------------------------------------------------------

SYNC “CompletionPoint”,-1

!----------------------------------------------------------------------------

! End of script

!----------------------------------------------------------------------------

Figure 9 -- SYNC.ACT

The GroupStart command begins a new group and names it "SimpleGroup". The group name is optional and will only be used for debug output purposes if an error occurs. Subsequent scripts that are executed prior to the GroupEnd command are considered to be within this group and can, therefore, synchronize on SYNC points.

The next command we issue, run "async.act", executes one instance (more can be specified) of the script async.act asynchronously and control returns immediately to the calling script. The instance of this script runs in its own namespace rather than in the namespace of the master script.

Before ending the group, we synchronously run the script sync.act with the command @sync.act. This causes the script to be run within the namespace of the master script and control does not return to the calling script until this script finishes executing.

Let’s look at the content of async.act in Figure 8 and sync.act in Figure 9.

Notice that both scripts have a synchronization point named "CompletionPoint" and the timeout value is set to -1 which will result in an infinite wait. The script async.act will be run and then control will return to master.act. However, async.act will halt execution at the SYNC command and wait for sync.act to reach the same SYNC point because they’re in the same group. This simple example just has each script create a different file in the same directory, write to it, delete it and then one script (async.act) waits for the other to complete before deleting the directory.

Conclusion
Developing a tool to exercise the native Windows API can be a useful addition to any Windows driver testing suite. When used in conjunction with Driver Verifier and Call Usage Verifier (CUV), both available from Microsoft and discussed elsewhere in this issue, you have a powerful combination of test tools for any driver under test. Don’t let unforeseen errors cripple the success of your driver!

User Comments
Rate this article and give us feedback. Do you find anything missing? Share your opinion with the community!
Post Your Comment

	Post Your Comments.
	Print this article.
	Email this article.