Libvirt of Vfio PCI passthrough
2015-08-18
First, prepare
1. Boot Start Item
/etc/default/grub file
Add Intel_iommu=on to the Grub_cmdline_linux parameter
2. Loading modules
Modprobe Vfio
Modprobe Vfio-pci
3. Prepare a virtual machine
E.g a virtual machine vfio_test
4. BIOS Open IOMMU Groups
Navigate through each BIOS screen using the "Arrow" keys and find the "IOMMU," "I/O Memory Management Unit," or "intel®v t-d "setting (usually located under the" Advanced "or" Chipset/northbridge/tylersburg Ioh/intel VT for Directed I/O Configu Ration "settings menu). Move the cursor over the setting selection box using the ' Arrow ' keys and press the ' page up ' or ' page DOWN ' or specified Key to select "Enabled."
Second, Iommu_group single equipment
1. Select the PCI device that needs to be passthrough, and its PCI address is 0000:0b:00.0
[root@host192 libvirt]# lspci | grep-i fibre
0b:00.0 Fibre channel:qlogic corp.isp2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)
2. Query the iommu_group that it is in, see Iommu_group is 19, and the entire group is a device 0000:0b:00.0
[root@host192 libvirt]# Readlink/sys/bus/pci/devices/0000\:0b\:00.0/iommu_group
.. /.. /.. /.. /kernel/iommu_groups/19
[root@host192 libvirt]# ls/sys/bus/pci/devices/0000\:0b\:00.0/iommu_group/devices/
0000:0b:00.0
3. Unbind the device under Iommu_group and unbind the device, the driver directory underneath the directory disappears.
Echo 0000:0b:00.0 >>/sys/bus/pci/devices/0000\:0b\:00.0/driver/unbind
Note: The driver directory under/sys/bus/pci/devices/0000\:0b\:00.0/will disappear after this instruction is executed
4. Bind device VendorID and DeviceID to new_id
[root@host192 libvirt]# lspci-n-S 0000:0b:00.0
0b:00.0 0c04:1077:2432 (Rev 03)
[root@host192 libvirt]# echo 1077 2432 >/sys/bus/pci/drivers/vfio-pci/new_id
5. Detect if the Iommu_group is bound successfully, you will find that the number of/dev/vfio under 19 this group
[root@host192 vfio]# Ls/dev/vfio
Vfio
6. Building XML, binding devices to virtual machines
[root@host192 ljl]# cat Net2.xml
<driver name= ' Vfio '/>
<source>
<address type= ' PCI ' domain= ' 0x0000 ' bus= ' 0x0b ' slot= ' 0x00 ' function= ' 0x0 '/>
</source>
[root@host192 ljl]# virsh attach-device vfio_test net2.xml--config
Device attached successfully
7. Start a virtual machine
[root@host192 ljl]# virsh start vfio_test
Domain Vfio_test started
Three, iommu_group under multiple devices (for example, two-port network card)
[root@host192 0000:02:00.0]# ls/sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
0000:02:00.0 0000:02:00.1
Multiple devices are similar to a single device, just need to unbind all the devices under the Iommu_group group, such as I want to passthrough0000:02:00.0 this device
Echo 0000:02:00.1 >>/sys/bus/pci/devices/0000:02:00.0/driver/ubind
Echo 0000:02:00.0 >>/sys/bus/pci/devices/0000:02:00.0/driver/ubind
Pay attention to the first 0000:02:00.1 to unbind off, because will 0000:02:00.0 to unbind, .../devices/0000:02:00.0/driver this directory will disappear, and then unbind down 0000:0 2:00.1 is obviously not possible.
This way you can passthrough off 02:00.0 of the device as before.
The binding process can be done through shell scripting:
#!/bin/sh
If [$#-ne 1]
then
echo "Usage: $ (basename $) pciaddr"
exit 1
fi
pciaddr=$1
Prefullpath= "/sys/bus/pci/devices/" $pciaddr
fullpath= $prefullpath "/iommu_group/devices"
Driverfullpath = $prefullpath "/driver/unbind" for
i in ' ls $fullpath '
do
if [$i! = $pciaddr] Then
# Unbind
echo $i >> $driverfullpath
# new_id
prdmsg= '/usr/sbin/lspci-n-S $pciaddr '
prdmsg= ' echo $ prdmsg | /usr/bin/sed ' s/://g ' | /usr/bin/awk-f "{print $4" "$ $}"
echo $prdmsg >/sys/bus/pci/drivers/vfio-pci/new_id
fi
done< c21/># unbind and new_id itself
echo $pciaddr >> $driverfullpath
prdmsg= '/usr/sbin/lspci-n-S $ Pciaddr '
prdmsg= ' echo $prdmsg |/usr/bin/sed ' s/://g ' |/usr/bin/awk-f ' {print $4 ' $ $ '
echo $prdmsg &G t;/sys/bus/pci/drivers/vfio-pci/new_id
Exit 0
Four, Iommu_group not only a number of equipment, but also bridge
[root@host192 ljl]# ls/sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/
0000:00:1c.0 0000:00:1c.4 0000:02:00.2 0000:03:00.0 0000:04:00.0
0000:00:1c.2 0000:02:00.0 0000:02:00.4 0000:03:00.1 0000:04:00.1
[root@host192 ljl]# lspci-s 0000:00:1c.0
00:1c.0 PCI Bridge:intel Corporation 82801JI (ich10family) PCI Express Root Port 1
[root@host192 ljl]# lspci-s 0000:00:1c.2
00:1C.2 PCI Bridge:intel Corporation 82801JI (ich10family) PCI Express Root Port 3
[root@host192 ljl]# lspci-s 0000:03:00.0
03:00.0 Ethernet controller:broadcomcorporation netxtreme II BCM5709 Gigabit Ethernet (rev 20)
The main difference is in the unbind and new_id, you need to iommu_group each device in their own driver/unbind to unbind off.
Like what:
For I in $ (ls/sys/kernel/iommu_groups/8/devices/)
Do
echo $i >>/sys/kernel/iommu_groups/devices/$i/driver/unbind
ven=$ (cat/sys/kernel/iommu_groups/8/devices/$i/vendor
dev=$ (cat/sys/kernel/iommu_groups/8/devices/$i/device
echo $ven $dev >/sys/bus/pci/drivers/vfio-pci/new_id
Done
V. Record of issues
1. Start a virtual machine error
Error:internal Error:process exited whileconnecting to MONITOR:2015-08-11T06:11:06.627255ZQEMU-KVM:-device Vfio-pci, Host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:vfio:failed to Open/dev/vfio/vfio:operation not permitted
2015-08-11t06:11:06.627315z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:vfio:failed to Setupcontainer for Group 19
2015-08-11t06:11:06.627331z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:vfio:failed to Get group19
2015-08-11t06:11:06.627351z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:device Initialization failed.
2015-08-11t06:11:06.627371z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:device ' Vfio-pci ' Couldnot be initialized
Workaround:
/etc/libvirt/qemu.conf file, add the "/dev/vfio/vfio" record in the CGROUP_DEVICE_ACL structure
2. Through a video card, the virtual machine does not boot
PCI 0000:08:00.0 is not assignable
Cause: QEMU cannot pass through devices that are not controlled by the ACS (Access control Service) By default, and you need to release QEMU's checksum if you want to pass through these devices
/etc/libvirt/qemu.conf
Turn on the Relaxed_acs_check = 1 option so that these devices can be used by virtual machine passthrough
3. Virtual machine does not boot, failed to set Iommu for container
Possible cause, vfio_iommu_type1.allow_unsafe_interrupts didn't turn on
After you turn on vfio_iommu_type1.allow_unsafe_interrupts, the virtual machine can start
You can also turn on vfio_iommu_type1.allow_unsafe_interrupts when the system is booted
4. Virtual machine does not boot,/DEV/VFIO/16 operation not permitted
/etc/libvirt/qemu.conf file, add the "/DEV/VFIO/16" record in the CGROUP_DEVICE_ACL structure
Vi. references
Http://www.redhat.com/archives/libvir-list/2013-March/msg00514.html
Http://www.linux-kvm.org/images/b/b4/2012-forum-VFIO.pdf
Https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and _administration_guide/chap-guest_virtual_machine_device_configuration.html#sect-pci_devices-pci_passthrough