An efficient data interaction mechanism between the application layer of domain0 and domainu Based on xen

There is a requirement in the project. The application layer of domain0 needs to regularly transmit a piece of data to the application layer of domainu (hvm windows). The original solution was to set up an HTTP server at the application layer of domainu, when the listener is listening on a port and data needs to be inserted, the domain0 application connects to the port and sends data through HTTP send. After sending the message, the application waiting for domainu to return a flag.

I accidentally saw this article "Communication Mechanism Analysis of fully virtualized hvm and semi-virtualized PV virtual platform", which introduced the data interaction mechanism between domainu and doamin0 user layer in the case of hvm. According to the article, try to design an interaction method: 1. add a custom Driver (datafront) to domainu, which provides interfaces to the application layer. When the application layer calls a specific system call or function, datafront triggers vmexit. hypervisor_callback corresponding to vmexit detects a shared memory area. If specific data is read to the domainu application layer. 2. Modify the qemu-DM daemon at the application layer in domain0, or write an application by yourself. When data needs to be inserted, write the data to the sharing page and send an event notification. The general process is like this, and the specific system will be considered later.

 

Reference: here

 

Communication Mechanism Analysis of fully virtualized hvm and semi-virtualized PV virtual platform

I. Communication Mechanism of the semi-virtualized PV virtual platform

Basically, all device drivers assume that they can directly access the hardware and have full control over the hardware. But it is impossible in this virtual machine system. A device usually serves multiple guest domains. To ensure management and secure access, xen adopts a separate Device Driver Model for device virtualization.

When guest domain is a quasi-virtualized virtual machine, the kernel of the virtual machine is modified, and it knows that it is not running on real hardware. The device is a device model separated by an xen Virtual Machine. frontend and backend work together to complete communication. The driver model of a separate device.

 

It can be seen that the front-end driver is located in the guest domain and is responsible for receiving the I/O processing requests from the guest domain and passing them to the backend driver ), and accept the processing result from the backend and return it to the guest domain. In domain0, the backend driver is responsible for receiving I/O processing requests from the front end, handing over the requests to the relevant drivers in domain0 for processing, and then returning the processing results to frontend. To complete the I/O operation of the guest domain.

Because the frontend and backend are in different operating systems, communication between them depends on the shared memory ring and event channel. A shared ring is a shared memory allocated by the front end, which is shared between the front end and the back end. There are two pairs of producer consumer pointers on the ring. Through the shared memory ring, the frontend and backend can put I/O requests into the ring and read from the ring, and the processing results of I/O requests can also be transmitted through the ring. The event channel allows the frontend and backend to send a confirmation message to the other party.

Xen uses the shared descriptor ring to implement the above operations. The principle is shown in.

 

Guest domain places the data transmission request on the I/Oring and updates the request producer pointer. xen retrieves the request and updates the consumer pointer. Similarly, when xen prepares the response data, it also places a descriptor on the I/Oring and updates the response producer pointer. The customer domain retrieves the response and submits the response to the application and updates the response consumer pointer.

It should be noted that the content on the I/oring is not the content of shared data, but the descriptor of the buffer where the shared data is located. This is because for high-speed DMA devices, it is inappropriate to use descriptors. The real data transmission is completed with Grant table. At the same time, requests are not processed in order:

Guest OS creates a unique identifier for each request, which will be copied in the response. The descriptor ring only limits the number of requests that can be processed, and does not specify the processing sequence. The process of who is first processed is determined when the data is mapped to the descriptor ring, this allows xen to reschedule the order of I/O operations for scheduling and priority. Xen uses the event channel as an asynchronous notification with an I/O descriptor entering the queue. Both request and response can put multiple descriptor items at the same time, an event notification is sent only when a threshold value is reached.

It details how two quasi-virtualized guest, domain1, and domain2 communicate on the xen Virtual Machine System. When app app1 in domain1 sends a packet to app app2 in domain2, it completes the following steps:

1. domain2 first calls send () to copy data from the user space to the kernel space;

2. The front-end of domain2 places the sending request on the sharing ring, which is completed by updating the producer sharing pointer;

3. the front-end will send an event (update, which triggers back-end to get the page containing app1 data) to domain0 through the event channel; back-end read request, obtain the physical address of the page and send it to the response driver;

4. xen removes the domain2 request from the sharing descriptor ring, moves the request consumer pointer, and schedules domain1;

5. The front end of domain1 fills in the IO loop with a pointer pointing to the free page in the grant table to receive the request;

6. when the back-end receives the data report, it first checks who the datagram belongs to, then removes the receiving request from the IO loop, and obtains the free page from the grant table, exchange the free pages and data pages, and send and receive events through the event channel;

7. frontend notification kernel, app2 receives data.

 

 

2. Guest domain is a fully virtualized Virtual Machine

When guest domian is a fully virtualized virtual machine, xen simulates the same abstract platform as the real machine. Guest domain does not know that it runs on the hypervisor. However, due to the introduction of hardware virtualization technology, for example, on the VT-x CPU, when the events defined in the Virtual Machine Control Structure vmcs (Virtual Machine Control Structure) structure are triggered, A vmexit occurs, switching from a non-root state to the root State, resulting in the hypervisor and the guest domain being switched out.

The vmcs structure consists of the customer status domain, host status domain, virtual machine execution control domain, vmexit control domain, vmentry control domain, and vmexit information domain. The Virtual Machine monitor mainly controls the execution behavior of virtual machines in non-root environments by configuring the virtual machine execution control domain. By setting the virtual machine execution control domain, vmm can set different virtual machine exit conditions for vmcs of different VMS to implement different virtualization policies for different VMS.

As shown in, when a fully virtualized Virtual Machine wants another target machine to send data, the process is as follows:

1. the kernel in guest domain executes the in/out command to trigger vmexit. The processor calls the vmexit processing function set by hypervisor.

2. hypervisor writes the specific information of the I/O commands to the I/O sharing page shared by DM of domainu, notifies domain0 through the event channel, and then the hypervisor blocks the virtual machine, and call the scheduling algorithm.

3. hypervisor restores the domain0 state and delivers the execution control to domain0.

4. In domain0, the registration callback function hypervisor_callback is executed first, and evtchn_do_upcall is called.

5. evtchn_do_upcall collects the number of I/O requests of virtual machines.

6. The execution control returns the user program State from the kernel state of xen0. The DM originally waited for the I/O request through a select system call. At this time, the scheduled DM will be returned once the request arrives.

7. by reading the I/O sharing page, DM identifies the access type of peripherals and calls the callback function when the corresponding Virtual Peripherals are initialized.

8. Based on different requests, the callback function of the Virtual Peripherals may write the status of the Virtual Peripherals back to the I/O sharing page, or a real data copy may occur. At last, DM still notifies hypervisor of processing through the event channel mechanism.

9. After the hypervisor is notified, block the corresponding guest domain of the request I/O.

At some point in the future, the guest domain can be scheduled again to continue running. The entire process is shown in: