Remember the resolution of Windows IO performance in a virtualized environment

Objective
With the development and progress of cloud computing technology and services, more and more customers choose to deploy the business to the cloud. However, due to the introduction of the virtualization layer, in the business deployment process often encounter IO problems, often difficult to debug. This article mainly introduces the use of tools such as Perf and Systemtap to help a hosted cloud customer Debug IO performance issues to analyze the performance of Windows IO in a virtual environment.

Problem arises
Once, hosted cloud customers built their own virtualized environments, created Windows R2 and Centos6.5 virtual machines on the same host, tested their random read performance with FIO, and the ioPS of Windows R2 were around 18K, And the ioPS of Linux can reach about 100K.
• FIO configuration for customer testing
[Global]
Ioengine=windowsaio
Direct=1
Iodepth=64
Thread=1
size=20g
Numjobs=1
[4k]
bs=4k
Filename=d:test.img
Rw=randread
Test results

Win_fio1
• Cloud Host IO stack

IO stack
In the cloud host environment, the entire IO stack is relatively long, involving the application layer/filesystem/block layer and the driver layer, the virtualization layer, the host OS filesystem/block layer and the driver layer in the guest OS. Because of the number of facets involved, so any one of the link problems will cause performance degradation, but also for the tracing of IO increased the difficulty.

From this information, we first ruled out the host file system and the block layer and the driver layer, because the same configuration, the Linux system is not a problem.
So the main focus is at two.
• Guest OS (Windows system)
•FIO Program
• file system/block Layer
• VirtIO block-driven virtual machines provide the guest OS with VirtIO block devices
• Qemu

How to eliminate the suspicion of QEMU?
For IOPS performance issues, it's easy to think of two possibilities:
• Io delay too high
• Device support IO queue is too short

In terms of queue problems, Linux and Windows virtual machines correspond to the same virtio block devices, then you need to confirm the delay problem.

How long did QEMU take to complete block IO?
Fortunately, Stefan Hajnoczi has added tracing features to QEMU, so it's easy to count the exact length of time that QEMU takes from receiving an IO request to completion.

From the above statistics, the average IO completion time in 130US, which temporarily ruled out the QEMU layer caused too high latency effect. In addition, if you are concerned about the overhead of this dynamic tracing, it is approximately 20% from the test observation.
Excluding queues and latency issues, the guest OS may be affected as well.
• VirtIO block-driven problem?
At least update to the latest stable version of the Virtio-win driver, the same problem still exists.
• Problems with the Windows file system/block layer?
The native Windows system did not make any configuration changes after confirmation.
Problems with the FIO test program

Why is there no problem with Fio on Linux?

Two Possibilities
In the performance troubleshooting process, it is always easy to get into the washout, often ask what is the problem? So all the factors that might affect it seem to have made no change. From experience, most performance problems can be divided into two possible ways:
•on CPU
•off CPU
To look at this problem again, after the basic troubleshooting IO delay problem, the corresponding problem has two possibilities:
• CPU is extremely busy, but most of the time is not doing IO processing;
• The CPU is often idle, and the corresponding is not primarily in the processing IO.
Note: It is not possible to exclude the effect of IO delay so far, because it excludes the potential impact of the QEMU block layer, but also the guest OS (this time ignoring the guest OS).
First look at the CPU consumption of the virtual machine during the test.
Top-h-P 36256

Win_fio1
From the point of view, the QEMU main thread CPU load has reached more than 90%, it seems to meet on the CPU class problem. In general, the best way to solve this kind of problem is to use the perf process to sample, and then generate a flame diagram, because it is a good idea to first see where the CPU is consumed.
Perf record-a-g-p 36256 sleep 20
To generate a flame diagram:

Win2008-bad
It is clear that the majority of CPU consumption is KVM operation, the most important consumption is vmx_handle_exit. (The real flame diagram is a vector diagram, which is easy to check with a browser). There are two main causes of Vmx_handle_exit:
• Access io Port (handle_pio)
• visit MMIO (handle_apic_access)
Since the KVM module is the majority, it is more desirable to understand the true behavior of KVM during testing, which can be achieved through another tool (Kvm_stat).

Kvm_pio
In addition to VM entry and VM exit events, the highest is Kvm_pio and Kvm_mmio, which shows that Windows does have a large number of IO ports and MMIO operations, which also validates the conclusions drawn on the flame map.
In virtualization, IO port or Mmio can cause VM exit, even heavy exit. If you need to improve performance, you will generally try to avoid this situation, at least avoid heavy Exit.

• Specifically access which IO ports and mmio result in VM Exit?

For this problem, the KVM module has added a lot of trace event, the above Kvm_stat also use these trace event, but not the specific trace event information printed out. In order to get trace-event information, there are many front-end tools, such as Trace-cmd, Perf, is a good choice.
• View trace event for all KVM modules
[Email protected]]# Trace-cmd list-e | grep KVM
Kvmmmu:kvm_mmu_pagetable_walk
Kvmmmu:kvm_mmu_paging_element
Kvmmmu:kvm_mmu_set_accessed_bit
Kvmmmu:kvm_mmu_set_dirty_bit
Kvmmmu:kvm_mmu_walker_error
Kvmmmu:kvm_mmu_get_page
Kvmmmu:kvm_mmu_sync_page
Kvmmmu:kvm_mmu_unsync_page
Kvmmmu:kvm_mmu_zap_page
Kvm:kvm_entry
Kvm:kvm_hypercall
Kvm:kvm_pio
Kvm:kvm_cpuid
Kvm:kvm_apic
Kvm:kvm_exit
Kvm:kvm_inj_virq
Kvm:kvm_inj_exception
Kvm:kvm_page_fault
Kvm:kvm_msr
Kvm:kvm_cr
Kvm:kvm_pic_set_irq
Kvm:kvm_apic_ipi
Kvm:kvm_apic_accept_irq
Kvm:kvm_eoi
Kvm:kvm_pv_eoi
Kvm:kvm_write_tsc_offset
Kvm:kvm_ple_window
Kvm:kvm_vcpu_wakeup
Kvm:kvm_set_irq
Kvm:kvm_ioapic_set_irq
Kvm:kvm_ioapic_delayed_eoi_inj
Kvm:kvm_msi_set_irq
Kvm:kvm_ack_irq
Kvm:kvm_mmio
The KVM module adds many trace event points, which only grab two of the--kvm:kvm_pio and Kvm:kvm_mmio.

Trace-cmd-pio-mmio

Primary access by statistical discovery:
• IO port is 0x608 and 0xc050;
• Mmio is 0xfee003xx
via QEMU info The Mtree command allows you to view the specific devices for IO Port 608, c050, and fee003xx respectively.
io Port
0000000000000608-000000000000060b (prio 0, RW): ACPI-TMR 000000000000c040-000000000000c07f (Prio 1, RW): Virtio-pci
Mmio
00000000FEE00000-00000000FEEFFFFF (prio 4096, RW): Icc-apic-container
c050 can be ignored, this is virtio block to do VM Exit.
So far, it can be determined that wnidows reads the ACPI Power Manager timer and accesses the APIC register, resulting in excessive VM exit generation and consumes a lot of CPU resources, so you can discuss two issues specifically:
1. How to reduce the VM exit caused by reading the ACPI PM Timer register;
2. How to reduce the VM exit that is caused by access to the APIC Mmio.

How can I reduce the VM Exit caused by reading the ACPI PM timer?
From the idea of virtualization layer optimization, reducing the VM exit caused by IO port usually considers whether paravirtulization can be used to replace full-virtualization for the purpose of seeing how Windows is doing this.
Starting with Windows 7, Microsoft has done a lot of virtualization enhancements to the Windows system in order to enable the Windows operating system to get better performance in Hyperv, including the Hyperv Timer that can be used here, This feature is similar to Kvmclock in Linux.
Judging from the current support situation:
• Windows 7
• Windows 7 SP1
• Windows Server R2
• Windows Server R2 sp1/sp2
• Windows 8/8.1/10
• Windows Server 2012
• Windows Server R2
These Windows systems all contain virtualization enhancements and more information on Microsoft's official website.
In 2014, Redhat engineers Vadim Rozenfeld and Peter Krempa added support for the Libvirt timer for QEMU and Hyperv respectively, so they can libvirt the timer directly through Hyperv.

<clock ...>

<timer name= ' hypervclock ' present= ' yes '/>

</clock>

In addition, the KVM also early support Hyperv Timer, but the customer's host kernel version does not support this feature, so you need to upgrade the customer Ucloud own maintenance of the kernel version.
• How to reduce APIC access and cause VM Exit?
Intel CPUs have also supported Apic-v, which is also upgraded to Ucloud's own maintenance kernel version.
Final effect

Win-fio-good

Win-good

Summarize
From this case, it can be seen that in a virtualized environment, compared to the physical environment, the poor performance of Windows Io is not necessarily a real IO path problem, and some virtualization performance issues can have a significant impact on IO performance.

Original address: http://blog.ucloud.cn/archives/2409https://my.oschina.net/u/3675312/blog/1529795

Remember the resolution of Windows IO performance in a virtualized environment