SR-IOV in Virtualization

There are many hardware acceleration technologies in the virtualization environment. These technical standards come from industry leaders or various organizations. But what will be enabled when the actual project is implemented? Which of the following features have significantly improved the performance? So what is the significance of the existence of these acceleration technologies?

No matter which solution, if you want to enable some acceleration functions, hardware support is required, which makes it difficult to implement it in some pre-project surveys or POC environments, after all, some requirements are very expensive and demanding, such as rdma. Compared with some technologies that require capital investment, sriov is undoubtedly relatively user-friendly and easy to implement. Today I will choose it to find out. This article describes how to use the Microsoft hyper-V environment.

Sriov is a single virtualization. Intel provided three-layer virtualization technology on the CPU and PCI bus to support the virtualization environment in the early days. They are:

1. Processor-based virtualization technology VT-x

2. I/O Virtualization Technology Based on PCI Bus VT-d

3. Network-based virtualization technology VT-C

Sriov is a branch of VT-D technology. To implement the sriov function, you must first support sriov, your motherboard MUST support VT-D technology (supporting VT-D naturally also supports sriov)

So what exactly does sriov use? What a considerable performance improvement can it bring to the virtualization platform? Let's take a look at the architecture diagram:

650) This. width = 650; "Title =" ic548927.png "alt =" wkiol1phnktsbmtdaaeoz1xd77e272.jpg "src =" http://s3.51cto.com/wyfs02/M02/3E/B3/wKioL1PHnkTSbmTdAAEoz1Xd77E272.jpg "/>

For example, explain the keywords one by one:

1. PF is a PCI function supported by physical NICs. pf can expand several VF

2. VF is a virtual "network card" or a virtual instance that supports sriov physical network card. It is displayed as an independent network card, each VF has its own exclusive PCI configuration area, and may share the same physical resource with other VF (share the same physical network port)

3. pf miniport driver: the PF driver works in the parent region of the hyper-V virtualization platform and is first loaded before VF.

4. VF miniport driver (VF driver) is used in the sub-region of the hyper-V virtualization platform, that is, guestos. Note that VF and PF are isolated, any results driven or executed by VF will not affect other VF or PF

5. The network interface card is the physical network card. After sriov is enabled, several vports are generated. What the physical Nic needs to do is to forward the traffic between the physical port and the vport.

6. Physical port is a physical network port. In sriov scenarios, physical port acts as an external network medium.

7. vports are abstract interfaces similar to physical network ports. They are mapped to each VF or PF for parentos or guestos.

The preceding architecture description shows that after sriov is enabled, the physical Nic interacts with the Virtual Machine (VF driver) through VF, and vice versa. In this way, the intermediate virtualization stack (vmm layer) can be skipped to achieve performance in almost pure physical environments. This is also the biggest value of sriov, he is different from the previous virtual machine traffic transfer through the simulation device and virtualization layer, so how much sriov can improve compared with the traditional environment, I will do an experiment:

Host OS: Windows Server 2012r2

Virtual Machine OS: Windows Server 2012r2

Server Model: Dell r720

NIC: Intel x520 Series

######################################## ######################################## ######

First, enable the sriov function in the server BIOS settings.

650) This. width = 650; "Title =" 3.png" style = "float: none;" alt = "wkiol1phps3ilxt1aazzjsaozdm624.jpg" src = "http://s3.51cto.com/wyfs02/M01/3E/BB/wKioL1PHpS3ilXt1AAZZJSAOZdM624.jpg"/>

After the sriov function is enabled on the physical machine, perform the following operations on the operating system. First, if hyper-V wants to use sriov, You need to modify it in two ways. One is a virtual switch, if you confirm that sriov (single virtualization) is enabled when creating a vswitch, note that once the vswitch is created, the sriov function cannot be modified. That is to say, if you forget to enable sriov, please delete it again

650) This. width = 650; "Title =" 4.png" style = "float: none;" alt = "wkiom1phpbscqaffaatqqnaicu480.jpg" src = "http://s3.51cto.com/wyfs02/M01/3E/BB/wKiom1PHpBSCQaFFAATQqNaicuU480.jpg"/>

After the vswitch enables sriov, it is necessary to operate on the virtual machine I tested, and enable sriov on the Virtual Machine's vnic, as shown in, which can be switched at any time.

650) This. width = 650; "Title =" 6.png" style = "float: none;" alt = "wkiol1phptdslly2aawapoumqo1_3.jpg" src = "http://s3.51cto.com/wyfs02/M02/3E/BB/wKioL1PHpTDSLLY2AAWAPOUMqO4403.jpg"/>

After confirming the above operations, you can use powershell to further confirm whether the system recognizes my settings and execute (get-vmhost) on the current host ). iovsupport or iovsupportreasons can be used to view the returned results. For object attributes in powershell, you can view the returned results using the pipeline "| GM ".

Also, as shown in, get-netadaptersriov is used to view what physical NICs support sriov on the current host, and from the returned results, my x520-2 Nic supports up to 62 VF.

Peripheral Component Interconnect Special Interest Group (specialized group for peripheral component interconnection), or pcisig, defines a maximum of 256 VF instances per device.

650) This. width = 650; "Title =" 7.png" style = "float: none;" alt = "wKiom1PHpBfCereuAAM-oVa_lNE338.jpg" src = "http://s3.51cto.com/wyfs02/M02/3E/BB/wKiom1PHpBfCereuAAM-oVa_lNE338.jpg"/>

After the sriov function is enabled correctly and successfully, I started this Test Virtual Machine sriov_2012. We can see that the current sriov is active under hyper-v, but the strange thing is that I found two IP addresses .. What's going on?

650) This. width = 650; "Title =" 8.png" style = "float: none;" alt = "wkiom1phpbmjbdxbaanfisclh9m014.jpg" src = "http://s3.51cto.com/wyfs02/M00/3E/BB/wKiom1PHpBmjbDXBAANfiScLh9M014.jpg"/>

Go to the Virtual Machine System to view the Device Manager and find an additional network card called "Intel (r) 82599 virtual function ", in fact, this intel x520 series Nic is based on the intel 82599 control chip, and the virtual function translation is a virtual function, that is, a virtual VF, it is displayed as a virtual network card in the virtual machine operating system, so I just saw two IP addresses

650) This. width = 650; "Title =" 9.png" style = "float: none;" alt = "wkiol1phptoqyvkgaaog7rsz75e943.jpg" src = "http://s3.51cto.com/wyfs02/M02/3E/BB/wKioL1PHpTOQyVKGAAOG7rSZ75E943.jpg"/>

There may be a small bug here, that is, I need to re-configure the IP address so that this virtual machine will not have two IP addresses. For example, the current normal test IP address shows (copy)

650) This. width = 650; "Title =" 10.png" style = "float: none;" alt = "wkiol1phptsq0eggaapgt_lo3xg727.jpg" src = "http://s3.51cto.com/wyfs02/M00/3E/BB/wKioL1PHpTSQ0eGGAAPGT_LO3Xg727.jpg"/>

After re-entering the IP address, it becomes normal again. That is to say, the original IP address is not directly mapped to my VF, and the current IP address has been restored to normal. There is only one IP address 6.6.6.0.

650) This. width = 650; "Title =" 11.png" style = "float: none;" alt = "wKiom1PHpBuA4kuQAARGEhine-Q270.jpg" src = "http://s3.51cto.com/wyfs02/M00/3E/BB/wKiom1PHpBuA4kuQAARGEhine-Q270.jpg"/>

You can also run the powershell command "get-netadaptersriovvf" to view the currently generated VF information.

650) This. width = 650; "Title =" 12.png" style = "float: none;" alt = "wkiom1phpbywpzobaak6nrwqxpg035.jpg" src = "http://s3.51cto.com/wyfs02/M01/3E/BB/wKiom1PHpBywPZObAAK6NrwqXpg035.jpg"/>

######################################## ######################################## ######

Next, a copy test is started. An ISO file is transmitted over the network. When sriov is enabled, the transmission speed is about 460 Mb/s.

650) This. width = 650; "Title =" 14.png" style = "float: none;" alt = "wkiol1phptj1_duzaarqr3rnedq374.jpg" src = "http://s3.51cto.com/wyfs02/M00/3E/BB/wKioL1PHpTjh7dUZAARQR3rNeDQ374.jpg"/>

While transferring files, I used the tool (burnintest) to pressurize the Virtual Machine CPU to simulate the actual situation as much as possible. The observed results are as follows: the performance monitor shows that the Minimum CPU usage is less than 2%, the maximum value is more than 11%, with a difference of about 9.5%.

650) This. width = 650; "Title =" 16.png" style = "float: none;" alt = "wKiom1PHpCLhV-IMAARcL9Pr1_U575.jpg" src = "http://s3.51cto.com/wyfs02/M01/3E/BB/wKiom1PHpCLhV-IMAARcL9Pr1_U575.jpg"/>

Disable the sriov function of the VM.

650) This. width = 650; "Title =" 17.png" style = "float: none;" alt = "wkiol1phpt3b_jiraavw.ibvne8855.jpg" src = "http://s3.51cto.com/wyfs02/M00/3E/BB/wKioL1PHpT3B_JirAAVg0iBvnE8855.jpg"/>

We can see that VF is missing, as shown in figure

650) This. width = 650; "Title =" 18.png" style = "float: none;" alt = "wkiom1phpctiy7btaandorsy5ro144.jpg" src = "http://s3.51cto.com/wyfs02/M02/3E/BB/wKiom1PHpCTiy7BTAANDorsY5Ro144.jpg"/>

Using powershell to confirm that VF is indeed far away from us ~

650) This. width = 650; "Title =" 19.png" style = "float: none;" alt = "wkiom1phpcxzvskoaamq2selmpw453.jpg" src = "http://s3.51cto.com/wyfs02/M01/3E/BB/wKiom1PHpCXzVskOAAMq2SelmPw453.jpg"/>

Copy the file again through the network, and it is still an ISO file (here I do not need to consider the cache factor, I will perform some copy operations before each copy to make it as full as possible ), the transmission rate is about 410 Mb/s.

650) This. width = 650; "Title =" Subtitle png" style = "float: none;" alt = "wkiol1phpucs846iaarzc2vrjmo478.jpg" src = "http://s3.51cto.com/wyfs02/M02/3E/BB/wKioL1PHpUCS846IAARzc2vRjMo478.jpg"/>
The vm cpu is pressurized during transmission. Observe the performance monitor results that the Minimum CPU load is less than 2%, and the maximum is close to 17%, with a difference of about 15%.

650) This. width = 650; "Title =" 21.png" style = "float: none;" alt = "wKioL1PHpUKw-FsCAAV6nzQcRXU291.jpg" src = "http://s3.51cto.com/wyfs02/M00/3E/BB/wKioL1PHpUKw-FsCAAV6nzQcRXU291.jpg"/>

Based on the above situation, compare the sriov function to enable and disable, copy the same ISO file and the same CPU pressurization method, and the results are as follows:

	Enable sriov	Disable sriov	Difference value
CPU usage	9.5%	15%	5.5%
Transmission Rate	460 MB/S	410 MB/S	50 MB/S

######################################## ######################################## ########

Through the above test, we can see that sriov enables or disables sriov, and the comparison still has some effect. My test environment is not rigorous enough, this is because many factors need to be considered in the actual production environment, such as disk I/O and Virtual Machine CPU configuration. However, even though it is rough, this data still has some reference value, about 6% of the CPU load and the speed difference of 50 MB/S I think it is quite impressive for any virtualization platform user who has a large number of concurrent requests, therefore, sriov is of great value to the current private cloud users.

Speaking of sriov, there is actually another "brother" who has to mention that it is VMQ (Virtual Machine Queue). My personal understanding is that sriov is more suitable for outgoing traffic, it optimizes the path of Data Interaction between virtual machines and physical machines, simplifies this process, and makes the performance almost pure physical environment. However, for inbound traffic, most of the processing work is done by the CPU, after external traffic comes in, the system needs to confirm which data is sent to which virtual machines. These routing and Distribution work is very laborious in scenarios with large traffic volumes. The VMQ function solves this problem precisely, it also requires support from physical NICs to route incoming traffic in advance and queue. The system directly allocates incoming traffic to the virtual machine to be received, improving performance. However, VMQ is based on the "first come, first served" mechanism, we recommend that you only use virtual machines with large volumes of access requests to avoid the impact on key services of those machines with only a small amount of incoming traffic.

There are many other hardware acceleration functions for virtualization. If you have any requirements in the future, we will share them with you.

This article from the "technology don't house" blog, please be sure to keep this source http://maomaostyle.blog.51cto.com/2220531/1439651