Image files used by QEMU: qcow2 and raw

Image files used by QEMU: qcow2 and raw

This article introduces qcow2 and raw, both of which are disk file formats used by QEMU (KVM) virtual machines. This article will compare and analyze their implementation principles, support features, and read/write efficiency, finally, we will introduce how to convert Disk Files in these two formats.

Basic Principles of qcow2

The qcow2 image format is a disk image supported by the QEMU simulator. It can also represent a fixed-size block device disk in the form of a file. Compared with images in raw format, raw images have the following features:

1. Smaller space usage, even if the file system does not support holes (holes );
2. Supports COW (copy-on-write). image files only reflect changes to the underlying disk;
3. Snapshots are supported. An image file can contain the history of multiple snapshots;
4. Zlib-based compression
5. Select AES encryption.

Qcow2 Image File Format header information

Each qcow2 file starts with a big-endian header. The structure is as follows:

Listing 1. qcow2 Header

 typedef struct QCowHeader {      uint32_t magic;      uint32_t version;      uint64_t backing_file_offset;      uint32_t backing_file_size;      uint32_t cluster_bits;      uint64_t size; /* in bytes */      uint32_t crypt_method;      uint32_t l1_size;      uint64_t l1_table_offset;      uint64_t refcount_table_offset;      uint32_t refcount_table_clusters;      uint32_t nb_snapshots;      uint64_t snapshots_offset;  } QcowHeader;

The following uses a 10G qcow2 file as an example to analyze the meaning of each field.

Listing 2. hexadecimal representation of the qcow2 File

# file 1.cow21.cow2: QEMU QCOW Image (v2), 10737418240 bytes0000000: 5146 49fb 0000 0002 0000 0000 0000 0000  QFI.............0000010: 0000 0000 0000 0010 0000 0002 8000 0000  ................0000020: 0000 0000 0000 0014 0000 0000 0003 0000  ................0000030: 0000 0000 0001 0000 0000 0001 0000 0000  ................0000040: 0000 0000 0000 0000 0000 0000 0000 0000  ................0000050: 0000 0000 0000 0000 0000 0000 0000 0000  ................0000060: 0000 0004 0000 0068 0000 0000 0000 0000  .......h........0000070: 0000 0000 0000 0000 0000 0000 0000 0000  ......................

The first four bits contain the characters Q, F, I, and 0xfb. The 5146 49fb In the instance is the magic field.

The next four bits contain the version number of the image file. In the instance, 0000 0002 is the version field, indicating that the qcow2 version is used.

Backing_file_offset occupies 8 bytes. In the instance, 0000 0000 0000 0000 indicates an offset starting from a file.

Backing_file_size indicates the length of a string that does not end with null. The value in the instance is 0000 0000. If the image file is copied at write time, it is the path of the original file.

Cluster_bits, 32-bit (0000 0010), describes how to map an image address to a local file. It determines how the low offset address in a cluster is indexed. Because the L2 table occupies a separate cluster and contains eight-byte table entries (entries), cluster_bits is only available in less than three places and serves as the index of the L2 table.

The next size, 8 bytes represents the size of the block device represented by the image file. The size of the instance is 0000 0002 8000 0000 bytes, that is, 10 Gb space.

If crypt_method is 1, AES encryption is used.

L1_size (0000 0014) and l1_table_offset (0000 0000 0003 0000: :) Give the L1 table size and offset respectively.

Refcount_table_offset returns the offset of the refcount table (0000 0000 0001 0000), while refcount_table_clusters describes the size of the refcount table in the unit of cluster (0000 0001 ).

Nb_snapshots shows the number of snapshots contained in the image (0000 0000), and snapshots_offset shows the offset from each snapshot to QCowSnapshotHeader (0000 0000 0000 ).

A typical qcow2 image file contains the following parts:

1. Header information mentioned above
2. L1 table
3. Refcount table
4. One or more refcount Blocks
5. Snapshot Header
6. L2 table
7. Data cluster

Level 2 search

In qcow2, the disk content is stored in the cluster (each cluster contains some 512-byte sectors ). To find the cluster where the given address is located, we need to find two tables, L1-> L2. The offset from the L1 table to the L2 table and the offset from the L2 table to the cluster;

Therefore, an address is divided into three parts based on cluster_bits (64-bit) settings, for example, cluster_bits = 12;

The 12-bit lower is the offset of a 4 kb cluster (the 12 power of 2 = 4 kb );

The next nine digits are L2 tables that contain 512 table items;

The remaining 43 digits represent the L1 Table offset.

To get the offset of a given address (64-bit:

1. Obtain the address of the L1 table from l1_table_offset in the Head field.
2. Use the first (64-l2_bits-cluster_bits) Address to index the L1 table
3. Get the address of the L2 table from the offset in the L1 table
4. Use the next l2_bits in the address to index the L2 table and obtain a 64-bit table item.
5. Obtain the cluster address from the offset in the L2 table.
6. Index the cluster with the remaining cluster_bits in the address to obtain the data block.

If the offset in both L1 and L2 tables is null, this region has not been allocated by the image file.

Note that the first two offset values of L1 and L2 tables are retained, which indicates 'copied' or 'compressed '.

Copy-on-Write image file

A qcow2 image can be used to save changes to another image file. Instead of modifying the original image file, it only records differences from the original image file, this image file is called a copy-on-write image. Although it is a separate file, most of its data comes from the original image. Only clusters with changes compared with the original image file will be recorded.

This is easy to implement. Simply record the original file path in the header information. When you need to read a cluster from a copy-on-write image file, first check whether the region has been allocated to the image file. If not, read the original file.

Snapshots

Snapshots are similar to Copy-On-Write files, but the difference is that snapshots are writable. The snapshot is the original file itself (internal snapshot ). It includes both the original file part before the snapshot is made, and it also contains the writable part.

Each snapshot contains the following header structure:

Listing 3. qcow2 snapshot Header

  typedef struct QCowSnapshotHeader {      /* header is 8 byte aligned */      uint64_t l1_table_offset;      uint32_t l1_size;      uint16_t id_str_size;      uint16_t name_size;      uint32_t date_sec;      uint32_t date_nsec;      uint64_t vm_clock_nsec;      uint32_t vm_state_size;      uint32_t extra_data_size; /* for extension */      /* extra data follows */      /* id_str follows */      /* name follows  */  } QcowSnapshotHeader;

Other features of qcow2

Qcow2 supports compression, which allows each cluster to use zlib for compression. It also supports encryption with a 128-bit AES key.

Create qcow2 and raw files and compare the two images

Use the QEMU-img software that comes with the qemu package to create the qcow2 file.

Listing 4. Creating qcow2 and raw files

$ qemu-img create -f qcow2 test.qcow2 10GFormatting 'test.qcow2', fmt=qcow2 size=10737418240 encryption=off cluster_size=65536 lazy_refcounts=off$ qemu-img create -f raw test.raw 10GFormatting 'test.raw', fmt=raw size=10737418240

The actual size and occupied space of the two formats are compared as follows:

Listing 5. Comparison of space occupied by qcow2 and raw files

$ Ll-sh test. * 200 K-rw-r -- 1 qiaoliyong 193 K May 6 10:29 test. qcow2 0-rw-r -- 1 qiaoliyong 10G May 6 10:28 test. raw [qiaoliyong @ localhost] $ stat test. raw file: "test. raw "Size: 10737418240 blocks: 0 IO blocks: 4096 common files [qiaoliyong @ localhost] $ stat test. qcow2 file: "test. qcow2 "Size: 197120 blocks: 400 IO blocks: 4096 common files

From the comparison, we can see that the size of the image file in qcow format is 197120 bytes, occupying 200 KB of space and 200 disk space. Raw files do not occupy disk space. They are empty files.

Raw format and qcow2 Conversion

The QEMU-img tool provided in the qemu package can be used for some common operations on image images.

To convert a raw file to qcow2, run the following command:

Qemu-img convert-f raw-O qcow2 test. raw test. raw. qcow2 [qiaoliyong @ localhost kimchi] $ ll-sh test. * 200 K-rw-r -- 1 qiaoliyong 193 K May 6 10:29 test. qcow2 0-rw-r -- 1 qiaoliyong 10G May 6 10:28 test. raw200K-rw-r -- 1 qiaoliyong 193 K May 6 10:44 test. raw. qcow2

Performance Comparison of Two formats of Files

Table 1. Performance Comparison of the three image formats using ide as the drive of the Virtual Disk

Cache =	Off	Writethrough	Writeback
Old qcow2 (0.10.5)	16: 52 min	28:58 min	6: 02 min
New qcow2 (0.11.0-rc1)	5: 44 min	9: 18 min	6: 11 min
Raw	Min	7: 24 min	6: 03 min

Table 2. Performance Comparison of Three image formats using virtio as the drive of a Virtual Disk

Cache =	Off	Writeback
Old qcow2 (0.10.5)	31: 09 min	8: 00 min
New qcow2 (0.11.0-rc1)	18: 35 min	8: 41 min
Raw	8: 48 min	7: 51 min

Summary

This article focuses on the format and features of the image file qcow2 used by the QEMU virtual machine, and compares it with the raw format image. Although qcow2 files have some performance loss compared with rRaw files (mainly reflected in the file increment, qcow2 files spend some time to allocate cluster files ), however, images in qcow2 format are smaller than Raw files. Files in qcow2 format increase only when the virtual machine actually occupies disk space, which can easily reduce the traffic consumed for migration, it is more suitable for cloud computing systems. It also has features that are not available in raw formats such as encryption, compression, and snapshots.

Ubuntu 12.04 cannot find the Qemu command

Install QEMU + efi bios on Arch Linux

QEMU translation framework and debugging tools

QEMU details: click here
QEMU: click here

This article permanently updates the link address: