Use vmkd to accelerate joint debugging of windbg virtual machines

Vmkd
Is a program that provides high speed kernel debugging support
For VMWare virtual machines. vmkd allows you to debug a VMware VM in
High Speed fashion, instead of using the much slower and lower
Bandwidth virtual serial port mechanic.

This page documents the general usage and installation of vmkd. There is, however, a post Series
With in-depth coverage of how vmkd works "under the hood" if you are interested in the Implementation Details of vmkd.

When you use vmkd to debug a VM, vmkd creates a named pipe on
Machine hosting the VM that you can connect to using the usual Kernel
Debugging Over named pipe support in windbg. However, unlike
Conventional VM kernel debugging, which is done by creating a virtual
Serial Port in the VM and exposing it to the host system as a named
Pipe, vmkd does not internally use a virtual serial port to communicate
With the kernel running in the VM. Instead, vmkd uses a high speed
Interface that takes advantage of the fact that the kernel is running
In a VM to enhance the performance and responsiveness of the kernel
Debugging experience.

Vmkd has presently only been tested against VMware Server 1.0.3 and 1.0.4.
It is designed in a fashion that is intended to be portable to future
VMware versions, however this forwards compatibility is fairly fragile
And may break on future releases. vmkd does not support Microsoft
Virtual Server or other virtualization alization products. Do not attempt to use
Vmkd with other virtualization programs or with a physical machine.

The main benefits of using vmkd instead of conventional serial port debugging are:

1. Responsiveness. vmkd provides a very low latency link between
   Kernel debugger and the VM if you are running the kernel debugging on
   The same computer hosting the VM. This means that most kernel debugger
   Commands will respond much quicker than with normal kernel debugging
   (Optional commands are typically fairly close to local kernel debugging
   (Lkd) speed, such! Process 0 0
   , Which typically returns in
   1-2 seconds or less even with 40-50 running processes when operating
   With vmkd). This improved response time even makes complex conditional
   Breakpoints on "warm" kernel code paths feasible!
2. Data transfer speed. vmkd can move data to and from a VM much
   Faster than the virtual serial port debugging mechanic. For example, I
   Typically converted Ed around ~ 200 kbps throughput while doing bulk memory
   Reads on a VM, far beyond that possible with a virtual serial port.
   Most of the overhead now remaining in terms of bulk data transfer is
   Reflective of design limitations of the Protocol that vmkd uses to talk
   To the kernel debugger client (dbgeng. dll). Note that 1394 can still
   Write physical memory dumps faster than vmkd, because 1394 KD can
   Essential tially DMA the target's physical memory loss ss the wire due
   Special support in the 1394 dbgeng KD protocol client. However, in most
   Other aspects, vmkd provides equivalent or superior performance to even
   1394 kD.
3. Processor usage. Normally, when you are kernel debugging
   Computer, the target is spinning on the kernel debugger I/O hardware
   (Such as the serial port or 1394 Controller). With a VM, This Is
   Participant ular problematic condition, as it causes the VM to monopolize one
   CPU with useless polling. vmkd allows the VM to sleep while waiting
   Input from the kernel debugger, eliminating the tendancy
   Conventional virtual serial port debugging to severely degrade overall
   System Performance on the host computer.

However, vmkd is not perfect. Because it was written without
Balance of either VMware or Microsoft, integration with the windows
Kernel and VMWare is a bit rough around the edges. Due to this, there
Are some steps that need to be followed to use vmkd. For some Kernel
Debugging tasks, it may be easier to just use virtual serial
Port debugging and live with the limitations of the virtual serial port
Than to set up a vmkd debugging session.

Additionally, vmkd is experimental software. Although I have
Attempted to test it on common deployments, it is possible that you
May encounter bugs that may crash your KD target Vm, or even
Vmware-vmx.exe process that you are using to host the target VM.

In order to use vmkd, you will need to do the following:

1. Download vmkd
   . The package provided des three distinct binaries, as well as documentation. The first of which isVmxpatch. dll
   ,
   Which is a DLL that is loaded into the vmware-vmx.exe process that is
   Hosting the VM that you wish to debug with vmkd. The second binary isVmxinject.exe
   .
   Vmxinject.exe is a simple tool that allows you to load vmxpatch. dll
   Into a target process (intended to be used on a vmware-vmx.exe
   Process). It takes a single command line argument, the process ID
   The vmware-vmx.exe process to load vmxpatch. DLL.

   The final binary encoded isKdvmware. sys
   (X86 and x64 flavors), which is a driver that needs to be installed in the guest operating system.

2. Create a virtual serial port on the VM that you want
   Debug, if you have not already. This is required so that you can enable
   Kernel debugging on the VM. You will then need to enable KD on
   Target VM (any parameters will work, so long as they are valid; I
   Recommend setting the VM up like you wowould for a normal virtual serial
   Port debugging session ).
3. Copy kdvmware. sys into the target Vm and install it. I wowould
   Recommend using the following command line to create the driver Service
   For it:
   

   SC create kdvmware type = kernel start = demand binpath = C:/Windows/system32/Drivers/kdvmware. sys displayname = kdvmware

   (C:/Windows/system32/Drivers/kdvmware. sys
   Corresponds
   The location where you have placed kdvmware. sys on the target VM). Do
   Not start the driver service yet. I recommend leaving the driver
   Service as manual start, as it will attempt to establish communications
   With vmxpatch. dll host-side and assume control of the kernel debugging
   Interface as soon as the driver is started.

4. Copy vmxpatch. dll and vmxinject.exe into a location on
   The computer hosting the VMware VM. Both files shoshould be placed in
   Same directory. When you start the VM that you want to debug, find
   PID corresponding to the vmware-vmx.exe instance hosting the VM. Then,
   Open a command prompt window, change to the directory in which you
   Placed vmxpatch. dll and vmxinject.exe, and issue the command:Vmxinject <vmware-vmx-process-ID>
   . This will load vmxpatch. DLL into the vmware-vmx.exe instance.

   If you are not certain what vmware-vmx.exe instance corresponds
   The VM that you want to kernel debug, Then I recommend using a tool
   Such as process explorer or windbg to examine either the command line
   Or Current Directory of each vmware-vmx.exe process until you find one
   That references the directory containing the VM you want to debug.

5. Load vmxpatch. DLL into the appropriate vmware-vmx.exe instance using vmxinject.exe as described above, if you have not already.
6. Start the kdvmware driver service on the target VM. It will attempt
   To connect to vmxpatch. dll, which shoshould have been loaded into
   Vmware-vmx.exe instance that is hosting the VM. If you did not load
   Vmxpatch. DLL into the correct vmware-vmx.exe instance, the VM will have
   Several "the operating system was not found." dialog boxes in
   VMware console, and the kdvmware. SYS driver will refuse to load. These
   Dialog Boxes are harmless and can be dismissed (internally, vmkd
   Repurposes the "operating system not found" VM monitor command
   Communicate with vmxpatch. dll). If you do encounter any such Dialog
   Boxes, then you will need to load vmxpatch. DLL into vmware-vmx.exe and
   Then start the kdvmware driver service in the target VM once more.

   If the driver successfully loads (and you do not get an error from
   Trying to start the driver, such as by "Net start kdvmware"), then it
   Has successfully established communications with vmxpatch. dll on
   Host.

7. Connect the kernel debugger to vmkd. vmkd creates a named pipe in the name "//./pipe/kdvmware _<Virtual Machine Directory Name>
   ", WhereVirtual Machine directory name
   Corresponds to the last Directory Component in the path to the VM's
   Configuration File. For example, if your VM is located in
   "C:/VMS/myvm", the pipe created will be of the name
   "//./Pipe/kdvmware_myvm". (Note that the pipe is named based off of the directory in which the VM is located and not the name of the VM
   .
   This approach was chosen to minimize the number of internal
   Vmware-vmx.exe functions that wowould need to be called, which in turn
   Improves compatibility with future VMware versions .)

   You shoshould be able to use the built in debugger named pipe support
   To communicate with the VM using the vmkd named pipe, just the same
   You wowould with a virtual serial port named pipe. Make sure that you
   Connect to the vmkd Named Pipe and not the virtual serial port named
   Pipe, as the kernel in the VM will no longer be flushing data from
   Virtual serial port.

For subsequent uses of vmkd, you will just need to use vmxinject.exe
To load vmxpatch. DLL into the correct vmware-vmx.exe instance and then
From there, start the kdvmware driver service once you are ready to cut
Kernel debugging for the VM over to vmkd.

Note that on x64 systems, it may be necessary to disable driver
Signature enforcement to load the kdvmware. SYS driver. Additionally, on
X64 systems, it will be necessary to disable patchguard
To use kdvmware. sys (the kdvmware driver rewrites the active Kernel
Debugger protocol module, which is unfortunately protected
Patchguard). This can be accomplished by attaching a debugger to
Virtual serial port at boot time so that patchguard is not enabled, and
Then after boot cutting the system over to vmkd. Due to these
Limitations, It is typically more convenient to use vmkd on 32-bit VMS.

The vmxpatch. dll module that is loaded into the vmware-vmx.exe
Process has various debug prints enabled in this release, which will be
Activated if a debugger is active in the vmware-vmx.exe process. If you
Experience difficulties while trying to connect to the vmkd Named Pipe
And synchronize with the target kernel, try attaching a debugger to
The vmware-vmx.exe process and examining the debug output. Additionally,
You can enable dbgeng KD protocol debugging with the Ctrl-d key
Combination (CTRL-alt-d For windbg). Keep in mind that watching
Vmxpatch. dll debug output with a Debugger Attached to vmware-vmx.exe
Will significantly lower both the throughput and responsiveness of
Vmkd connection.

 

From: http://www.nynaeve.net /? Page_id = 168