Five RDMA Application Models of VMware vSphere

RDMA (remote direct memory access) is a remote direct memory access technology. RDMA can transmit memory data between two hosts without the intervention of the operating system. RDMA technology features low IO latency, high bandwidth performance, and low host CPU load. VSphere is the core product of VMware virtualization. Its RDMA technology can effectively meet the requirements of Virtual Machine migration and distributed storage. This article mainly introduces five Application Models of RDMA in vSphere.

Full-function VM DirectPath

Full-function VM DirectPath is directly transparently transmitted to a virtual machine by the Virtual Machine VMM/Hypervisor. PCI devices refer to HCA adapters, which can be based on IB, RoCe, or iWRAP. A physical device is used independently by a virtual machine, and cannot be used between virtual machines. The virtual machine is shared with VMM/Hypervisor.

The RDMA performance of the virtual machine is very good because it is directly transmitted through VMM/Hypervisor, which is very similar to the RDMA performance of the physical machine. The disadvantage of this solution is that many vSphere software functions are unavailable, such as FT, Snapshot, Suspend, and vMotion.

Vf vm DirectPath SR-IOV

SR-IOV VF solves the problem that a single PCI device cannot be shared by multiple virtual machines under Full-function. SR-IOV requires support for adapter and/VMM/Hypervisor collaboration. An adapter can register multiple independent interrupts (PF and VF) on the chipset. Each interrupt corresponds to an independent PCI device on the VMM/Hypervisor. VMM/Hypervisor transparently transmits each PCI device (VF) to one or more virtual machines. PF is allocated to VMM/Hypervisor to manage VF.

The same is the VMM/Hypervisor passthrough. The RDMA performance of the virtual machine is very good, very close to the RDMA performance of the physical machine. In addition, thanks to the hardware support of the adapter, The VMM/Hypervisor can also flexibly manage QOS. The disadvantage of this solution is the same as that of Full-function. Many software functions based on vSpher are incompatible.

RoCE Simulation Based on 10GbE VM DirectPath:

VMM/Hypervisor allocates the 10GbE adapter to the virtual machine through passthrough through the PCI device. The virtual machine runs the RXE driver to implement the RoCE function, which can communicate with the virtual machine running the RXE driver or the RoCE hardware adapter. This is an RDMA method that is simulated on a virtual machine with poor performance. Because it is VM DirectPath, many vSphere performance cannot be used.

Semi-virtualization-based RoCE Simulation

The virtual machine also implements the software simulation RoCE function by installing the RXE driver. Unlike the RoCE Simulation Based on the VM DirectPath, the virtual machine Nic uses the semi-virtualization solution (vmxnet3 ). This scheme can be used with a server that does not need to rely on physical hardware. The server does not have an adapter that supports the RDMA function in a timely manner, and the two virtual machines can also perform RDMA traffic.

The performance of the semi-virtualization network is good, between VMDirectPath and full simulation. The full simulation performance of the RDMA Based on semi-virtualization is poor, and the advantage is that vSphere functions are available.

Infiniband-based semi-RDMA Virtualization (vRDMA)

The server uses the Infiniband physical adapter, and the VMM/Hypervisor provides a virtual RDMA solution for virtual machines. The Virtual Machine RDMA driver is implemented based on Infiniband Verbs. VMM/Hypervisor maps the memory of the virtual machine to the memory of the RDMA physical adapter. The virtual machine memory access can be 0-copied through DMA.

Summary

There are three main methods for virtualization application RDMA: simulation, semi-virtualization, and passthrough. The passthrough mode has the best performance, better semi-virtualization performance, and average simulation mode performance. The PCI-E-based passthrough mode allows multiple virtual machines to share one physical device, the performance of Virtual Machine RDMA is very close to the physical performance, and can implement QOS Management in VMM/Hypervisor. The semi-virtualization solution can achieve considerable performance without affecting a large number of vSphere software functions (such as FT and vMotion ). Both the passthrough mode and the semi-virtualization mode must work with the RDMA physical adapter.