How to use the VDMDBG function in Windows NT, Windows 2000, and Windows XP

Summary

Even if you write programs in a 32-bit environment, you often encounter situations where you are dealing with 16-bit applications. In Windows NT, 16-bit programs are run in virtual DOS machines (VDM). VDMDBG.dll contains a number of useful functions for handling 16-bit programs, which are part of the Platform SDK.

The functions in vdmdbg provide a good way to enumerate, create, and terminate 16-bit processes (tasks) in VDM. This article describes how to use these functions in Windows NT, Windows 2000, and Windows XP.

If you need to use the VDMDBG function, your project must link VDMDBG.lib.

The vdmdbg function mentioned below is only a subset. Functions that are not mentioned are related only to the debugger.

Enumerating VDMs

The Vdmenumprocesswow () function provides an easy way to enumerate all the VDMS that are running Windows tasks. These vdms contain WowExec.exe tasks. DOS VDMs is not listed in the enumeration. The declaration of the function is as follows:INT WINAPI VDMEnumProcessWOW( PROCESSENUMPROC fp, LPARAM lparam );

The return value of the function is the number of VDMS currently running, or the number of enumerations before the enumeration is terminated. FP is a pointer to a callback function. This function is invoked once for the enumerated VDM. LParam is a user-defined value that is passed to the callback function.

Processenumproc's statement reads as follows:

typedef BOOL ( WINAPI *PROCESSENUMPROC )(
DWORD dwProcessId,
DWORD dwAttributes,
LPARAM lpUserDefined
);

This function returns TRUE to stop the enumeration and returns FALSE to continue the enumeration. Dwprocessid is the NTVDM.exe process ID. You will need this ID when other VDM functions are invoked (these functions are mentioned below).

Enumerate 16-bit Windows tasks

You can enumerate tasks in a particular VDM with Vdmenumtaskwow () and VDMEnumTaskWOWEx (). The difference between the two is that VDMEnumTaskWOWEx () provides more information for the callback function. You can invoke these task enumeration functions only by using the VDMS returned by Vdmenumprocesswow (). Using DOS VDMs is meaningless because the DOS application runs in its own VDM. The evil declarations of these two functions are as follows:

INT WINAPI VDMEnumTaskWOW( DWORD dwProcessId, TASKENUMPROC fp,LPARAM lparam );
INT WINAPI VDMEnumTaskWOWEx( DWORD dwProcessId, TASKENUMPROCEX fp,
LPARAM lparam );

The return value of the two functions is the number of tasks currently running in the specified VDM or the enumerator before the end of each inning. Dwprocessid is the evil process ID of VDM. FP is a pointer to a callback function. This function is called for each task that is enumerated. LPARAM is a user-defined value passed to the callback function.

The definitions of Taskenumproc and Taskenumprocex are as follows:

typedef BOOL ( WINAPI *TASKENUMPROC )(
DWORD dwThreadId,
WORD hMod16,
WORD hTask16,
LPARAM lpUserDefined
);
typedef BOOL ( WINAPI *TASKENUMPROCEX )(
DWORD dwThreadId,
WORD hMod16,
WORD hTask16,
PSZ pszModName,
PSZ pszFileName,
LPARAM lpUserDefined
);

These functions return TRUE to stop the enumeration and return FALSE to continue the enumeration. You can terminate a task with hTask16 in a call.