Libvirt of Vfio PCI passthrough

Libvirt of Vfio PCI passthrough

2015-08-18

First, prepare

1. Boot Start Item

/etc/default/grub file

Add Intel_iommu=on to the Grub_cmdline_linux parameter

2. Loading modules

Modprobe Vfio

Modprobe Vfio-pci

3. Prepare a virtual machine

E.g a virtual machine vfio_test

4. BIOS Open IOMMU Groups

Navigate through each BIOS screen using the "Arrow" keys and find the "IOMMU," "I/O Memory Management Unit," or "intel®v t-d "setting (usually located under the" Advanced "or" Chipset/northbridge/tylersburg Ioh/intel VT for Directed I/O Configu Ration "settings menu). Move the cursor over the setting selection box using the ' Arrow ' keys and press the ' page up ' or ' page DOWN ' or specified Key to select "Enabled."

Second, Iommu_group single equipment

1. Select the PCI device that needs to be passthrough, and its PCI address is 0000:0b:00.0

[root@host192 libvirt]# lspci | grep-i fibre

0b:00.0 Fibre channel:qlogic corp.isp2432-based 4Gb Fibre Channel to PCI Express HBA (rev 03)

2. Query the iommu_group that it is in, see Iommu_group is 19, and the entire group is a device 0000:0b:00.0

[root@host192 libvirt]# Readlink/sys/bus/pci/devices/0000\:0b\:00.0/iommu_group

.. /.. /.. /.. /kernel/iommu_groups/19

[root@host192 libvirt]# ls/sys/bus/pci/devices/0000\:0b\:00.0/iommu_group/devices/

0000:0b:00.0

3. Unbind the device under Iommu_group and unbind the device, the driver directory underneath the directory disappears.

Echo 0000:0b:00.0 >>/sys/bus/pci/devices/0000\:0b\:00.0/driver/unbind

Note: The driver directory under/sys/bus/pci/devices/0000\:0b\:00.0/will disappear after this instruction is executed

4. Bind device VendorID and DeviceID to new_id

[root@host192 libvirt]# lspci-n-S 0000:0b:00.0

0b:00.0 0c04:1077:2432 (Rev 03)

[root@host192 libvirt]# echo 1077 2432 >/sys/bus/pci/drivers/vfio-pci/new_id

5. Detect if the Iommu_group is bound successfully, you will find that the number of/dev/vfio under 19 this group

[root@host192 vfio]# Ls/dev/vfio

Vfio

6. Building XML, binding devices to virtual machines

[root@host192 ljl]# cat Net2.xml

<driver name= ' Vfio '/>

<source>

<address type= ' PCI ' domain= ' 0x0000 ' bus= ' 0x0b ' slot= ' 0x00 ' function= ' 0x0 '/>

</source>

[root@host192 ljl]# virsh attach-device vfio_test net2.xml--config

Device attached successfully

7. Start a virtual machine

[root@host192 ljl]# virsh start vfio_test

Domain Vfio_test started

Three, iommu_group under multiple devices (for example, two-port network card)

[root@host192 0000:02:00.0]# ls/sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/

0000:02:00.0 0000:02:00.1

Multiple devices are similar to a single device, just need to unbind all the devices under the Iommu_group group, such as I want to passthrough0000:02:00.0 this device

Echo 0000:02:00.1 >>/sys/bus/pci/devices/0000:02:00.0/driver/ubind

Echo 0000:02:00.0 >>/sys/bus/pci/devices/0000:02:00.0/driver/ubind

Pay attention to the first 0000:02:00.1 to unbind off, because will 0000:02:00.0 to unbind, .../devices/0000:02:00.0/driver this directory will disappear, and then unbind down 0000:0 2:00.1 is obviously not possible.

This way you can passthrough off 02:00.0 of the device as before.

The binding process can be done through shell scripting:

#!/bin/sh
 
If [$#-ne 1]
then
   echo "Usage: $ (basename $) pciaddr"
   exit 1
fi
 
pciaddr=$1
Prefullpath= "/sys/bus/pci/devices/" $pciaddr
fullpath= $prefullpath "/iommu_group/devices"
Driverfullpath = $prefullpath "/driver/unbind" for
 
i in ' ls $fullpath '
do
   if [$i! = $pciaddr] Then
       # Unbind
  echo $i >> $driverfullpath
 
       # new_id
       prdmsg= '/usr/sbin/lspci-n-S $pciaddr '
       prdmsg= ' echo $ prdmsg | /usr/bin/sed ' s/://g ' | /usr/bin/awk-f "{print $4" "$ $}"
       echo $prdmsg >/sys/bus/pci/drivers/vfio-pci/new_id
   fi
done< c21/># unbind and new_id itself
echo $pciaddr  >> $driverfullpath
prdmsg= '/usr/sbin/lspci-n-S $ Pciaddr '
prdmsg= ' echo $prdmsg |/usr/bin/sed ' s/://g ' |/usr/bin/awk-f ' {print $4 ' $ $ '
echo $prdmsg &G t;/sys/bus/pci/drivers/vfio-pci/new_id
 
Exit 0

Four, Iommu_group not only a number of equipment, but also bridge

[root@host192 ljl]# ls/sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/

0000:00:1c.0 0000:00:1c.4 0000:02:00.2 0000:03:00.0 0000:04:00.0

0000:00:1c.2 0000:02:00.0 0000:02:00.4 0000:03:00.1 0000:04:00.1

[root@host192 ljl]# lspci-s 0000:00:1c.0

00:1c.0 PCI Bridge:intel Corporation 82801JI (ich10family) PCI Express Root Port 1

[root@host192 ljl]# lspci-s 0000:00:1c.2

00:1C.2 PCI Bridge:intel Corporation 82801JI (ich10family) PCI Express Root Port 3

[root@host192 ljl]# lspci-s 0000:03:00.0

03:00.0 Ethernet controller:broadcomcorporation netxtreme II BCM5709 Gigabit Ethernet (rev 20)

The main difference is in the unbind and new_id, you need to iommu_group each device in their own driver/unbind to unbind off.

Like what:

For I in $ (ls/sys/kernel/iommu_groups/8/devices/)

Do

echo $i >>/sys/kernel/iommu_groups/devices/$i/driver/unbind

ven=$ (cat/sys/kernel/iommu_groups/8/devices/$i/vendor

dev=$ (cat/sys/kernel/iommu_groups/8/devices/$i/device

echo $ven $dev >/sys/bus/pci/drivers/vfio-pci/new_id

Done

V. Record of issues

1. Start a virtual machine error

Error:internal Error:process exited whileconnecting to MONITOR:2015-08-11T06:11:06.627255ZQEMU-KVM:-device Vfio-pci, Host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:vfio:failed to Open/dev/vfio/vfio:operation not permitted

2015-08-11t06:11:06.627315z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:vfio:failed to Setupcontainer for Group 19

2015-08-11t06:11:06.627331z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:vfio:failed to Get group19

2015-08-11t06:11:06.627351z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:device Initialization failed.

2015-08-11t06:11:06.627371z QEMU-KVM:-devicevfio-pci,host=0e:00.0,id=hostdev0,bus=pci.2,addr=0x6:device ' Vfio-pci ' Couldnot be initialized

Workaround:

/etc/libvirt/qemu.conf file, add the "/dev/vfio/vfio" record in the CGROUP_DEVICE_ACL structure

2. Through a video card, the virtual machine does not boot

PCI 0000:08:00.0 is not assignable

Cause: QEMU cannot pass through devices that are not controlled by the ACS (Access control Service) By default, and you need to release QEMU's checksum if you want to pass through these devices

/etc/libvirt/qemu.conf

Turn on the Relaxed_acs_check = 1 option so that these devices can be used by virtual machine passthrough

3. Virtual machine does not boot, failed to set Iommu for container

Possible cause, vfio_iommu_type1.allow_unsafe_interrupts didn't turn on

After you turn on vfio_iommu_type1.allow_unsafe_interrupts, the virtual machine can start

You can also turn on vfio_iommu_type1.allow_unsafe_interrupts when the system is booted

4. Virtual machine does not boot,/DEV/VFIO/16 operation not permitted

/etc/libvirt/qemu.conf file, add the "/DEV/VFIO/16" record in the CGROUP_DEVICE_ACL structure

Vi. references

Http://www.redhat.com/archives/libvir-list/2013-March/msg00514.html

Http://www.linux-kvm.org/images/b/b4/2012-forum-VFIO.pdf

Https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Virtualization_Deployment_and _administration_guide/chap-guest_virtual_machine_device_configuration.html#sect-pci_devices-pci_passthrough