Understanding OpenStack & Ceph (4): Ceph's underlying data structure [Pool, Image, Snapshot, Clone]

This series of articles will delve into the integration of Ceph and Ceph and OpenStack:

(1) Installation and deployment

(2) Ceph RBD interfaces and tools

(3) Ceph physical and logical structure

(4) Ceph's underlying data structure

(5) The implementation of Ceph and OpenStack integration

(6) TBD

1 Pool (pools)

The concept of pool has been said before, Ceph supports a rich pool operation, including:

List, create and delete poolceph OSD pool Create {pool-name} {pg-num} [{pgp-num}] [replicated] [crush-ruleset-name]ceph OSD Pool Create { Pool-name} {Pg-num} {pgp-num} erasure [Erasure-code-profile] [crush-ruleset-name]ceph OSD Pool Delete {Pool-name} [{ Pool-name}--yes-i-really-really-mean-it]qos support: Ceph OSD pool Set-quota {Pool-name} [max_objects {obj-count}] [Max_ bytes {bytes}] snapshot creation and deletion: Ceph OSD pool Mksnap {pool-name} {snap-name}ceph osd pool Rmsnap {pool-name} {snap-name} metadata modified Ceph OS D Pool set {Pool-name} {key} {value} Sets the number of copies of the object (note that the number of copies includes the object itself) Ceph OSD Pool set {poolname} size {Num-replicas} copy in degraded mode Copies of Ceph OSD Pool Set data min_size 2

2 Juan (image) 2.1 Image users See

The Image corresponds to the LVM Logical Volume, which will be striped into N sub-blocks, and each block of data will be stored as an object in the RADOS object store (simple block devicees). Like what:

#创建 the size of the RBD Image named ' MyImage ', by default it is striped to a size of 4MB objects RBD create mypool/myimage--size 102400#同样是 100 MB size of RBD Image, but it is striped to a 8MB size of 13 objects RBD create Mypool/myimage--size 102400--order 23

#将 image mount to a Linux host is called a DEIVCE/DEV/RBD1
RBD Map Mypool/myimage

#向/DEV/RBD1 Write Data
DD If=/dev/zero of=/dev/rbd1 bs=1047586 count=4# Delete IMAGERBD rm mypool/myimage

See 2.2 Image of the CEPH system

Let's take a look at some of the internal information of image.

(1) Create a new object

First create an image in an empty pool with a

[Email protected]:~# RBD create-p pool100 image1--size 102400--image-format 2[email protected]:~# RBD List Pool 100image1 

At this time I saw some more objects in the pool:

[Email protected]:~# rados-p pool100 lsrbd_directoryrbd_id.image1rbd_header.a89c2ae8944a

As can be seen from the name, these objects do not store image data, but are metadata information such as Id,header. where all the IDs and name information for image are saved in Rbd_directory:

[Email protected]:~# rados-p pool100 listomapvals rbd_directoryid_a89c2ae8944avalue: ( bytes): 0000:06 00 00 6d.                   image1name_image1value: ( bytes): 0000:0c 00 00 00 61 38 39 63 32 61 65 38 39 3 4:    .... a89c2ae8944a

And Rbd_header is storing the metadata for the Image:

[Email protected]:~# rados-p pool100 listomapvals rbd_header.a89c2ae8944afeaturesvalue: (8 bytes): 0000:01 00 0  0 xx xx                         : ... object_prefixvalue: ( bytes): 0000:15 (2e), 5f (38)----61 A : .... rbd_data.a890010:63: c2ae8944aordervalue: (1 bytes): 0000:16 :. siz Evalue: (8 bytes): 0000:00 xx (xx): ... Snap_seqvalue: (8 bytes): 0000:00 00 00 00 00 XX:           .....

This information is the source of information for the following command:

[Email protected]:~# RBD-P pool100 info image1rbd image ' Image1 ': in    25600 objects    order 2 2 (4096 KB objects)    block_name_prefix:rbd_data.a89c2ae8944a    format:2 features:layering 

It can also be seen that the name prefix of the image's data object is rbd_header.a89c2ae8944a. For a new image, because there is no data object, its actual storage space is only a very small amount of space that the metadata object occupies.

(2) Write data to the object 8MB (an object in the pool is 4MB)

[Email protected]:~# RBD map pool100/image1[email protected]:~# RBD Showmappedid pool    image  snap device    1  pool100 image1-    /dev/
[Email protected]:~# dd If=/dev/zero of=/dev/rbd1 bs=1048576 count=88+0 Records in8+0 records out8388608 bytes (8.4 MB) CO Pied, 0.316369 s, 26.5 MB/s

Look at the objects in the pool:

[Email protected]:~# rados-p pool100 lsrbd_directoryrbd_id.image1rbd_ data.a89c2ae8944a.0000000000000000rbd_data.a89c2ae8944a.0000000000000001rbd_header.a89c2ae8944a 

You can see that there are 2 more 4MB of object. Continue to see the OSD where the first object is located:

[Email protected]:~# ceph OSD Map pool100 rbd_data.a89c2ae8944a.0000000000000000object ' Rbd_ data.a89c2ae8944a.0000000000000000 ', pg 7.df059252 (7.52), Up ([8,6,7], P8) acting ([8,6,7], P8)

The PG ID is 7.52, the main OSD is 8, and the OSD is 6 and 7. View the node for OSD 8 as Ceph3:

[Email Protected]:/data/osd2/current/7. 52_head# ceph OSD tree# ID    weight    type name    up/    down Reweight-1    0.1399    root Default    -4    0.03998        host ceph35    0.01999            Osd.5    Up    1    8    0.01999            Osd.8    up    1  

Log in to Ceph3, view the/VAR/LIB/CEPH/OSD directory, and see the Ceph-8 directory:

[Email protected]:/var/lib/ceph/osd# ls-ltotal 0lrwxrwxrwx 1 root root  9 Sep 02:59 ceph-5-/data/< c26>osdlrwxrwxrwx 1 root root Sep 08:22 ceph-8-/data/osd2  

You can see two data files at the beginning of the 7.52 directory:

[Email protected]:/data/osd2/current# Find. -name ' *a89c2ae8944a* './7.5c_head/rbd\uheader.a89c2ae8944a__head_36b2dadc__7./7.52_head/rbd\ Udata.a89c2ae8944a.0000000000000001__head_9c6139d2__7./7.52_head/rbd\udata.a89c2ae8944a.0000000000000000__ Head_df059252__7   

Visible:

(1) RBD image is a simple block device that can be directly mount to the host and become a device that users can write directly to binary data.

(2) The data of image is saved as a number objects in the RADOS object store.

(3) The data space of image is thin provision, which means that Ceph does not pre-allocate space, but instead waits until the data is actually written to allocate space according to object.

(4) Each data object is saved as multiple copies.

3 Snapshots (snapshot) 3.1 Snapshot users see

A snapshot of the RBD image (snapshot) is a read-only copy of the image at a specific point in time (a snapshot is a read-only copy of the state of an image at A particular p Oint in time.). Note that you need to stop I/O before doing snapshot, and if the image contains a file system, the system ensures that the file system is in a contiguous state.

Users can use the RBD tool or other API to manipulate snapshot:

RBD create-p pool101--size 102400 image1--format 2 #创建 image
RBD snap Create Pool101/[email protected] #创建snapshotRBD snap ls pool101/image1 #列表RBD snap Protect Pool101/[email Protected] #保护RBD snap unprotect Pool101/[email protected] #去保护RBD snap rollback pool101/[email protected] #回滚sna Pshot to image. Note that this is a time-consuming operation, RBD will display a progress bar RBD snap rm pool101/[email protected] #删除
RBD snap purge pool101/image1 all snapshot for #删除 image
RBD clone Pool101/[email protected] Image1snap1clone1 #创建 clone
RBD children Pool101/[email protected] #列表它的所有 clone

See 3.2 Snapshot's Ceph system

Let's take a look at the inner workings of snapshot.

(1) Create Image1, write 4MB of data, then create a SNAPSHOT:RBD snap create Pool100/[email protected]

(2) Ceph does not create a new object in the pool, meaning that there is no storage space allocated to SNAP1 to create the data objects.

[Email protected]:~# RBD Map Pool101/image1
[Email protected]:~# RBD showmapped
ID Pool Image Snap device
1 pool100 image1-/DEV/RBD1
2 pool101 Image1-/dev/rbd2

[Email protected]:~# dd if=/dev/sda1 of=/dev/rbd2 bs=1048576 count=4
4+0 Records in
4+0 Records out
4194304 bytes (4.2 MB) copied, 0.123617 s, 33.9 MB/s
[Email protected]:~# rados-p pool101 ls
Rbd_directory
rb.0.fc9d.238e1f29.000000000000
Image1.rbd

[Email protected]:~# RBD snap Create Pool101/[email protected]

[Email protected]:~# RBD snap ls pool101/image1
Snapid NAME SIZE
Ten Snap1 102400 MB

[Email protected]:~# rados-p pool101 ls
Rbd_directory
rb.0.fc9d.238e1f29.000000000000
Image1.rbd

(3) Ceph instead increased the snapshot information to Rbd_header. {image_id} object

[Email protected]:~# rados-p pool101 listomapvals rbd_header.a9262ae8944a
snapshot_0000000000000006

Value: (bytes):
0000:03 01 44 00 00 00 06 00 00 00 00 00 00 00 05 00:.. D.............
0010:00 6e 61 70 31 00 00 00 00 19 00 00 00 01:.. Snap1 .....
0020:00 (XX) to 1c, XX, FF FF FF: ..........
0030:FF FF FF FF (f) F/FE FF FF FF FF FF: .......
0040:ff 00 00 00 00 00 00 00 00 00: .....

(4) write the 4MB data to the Image1 (actually overwrite the data in the first object) and find a 4MB file in the Data directory:

[Email protected]:/data/osd2/current/8.2c_head# ls/data/osd/current/8.3e_head/-L
Total 8200
-rw-r--r--1 root root 4194304 Sep 03:25 rb.0.fc9d.238e1f29.000000000000__a_ae14d5be__8
-rw-r--r--1 root root 4194304 Sep 03:25 rb.0.fc9d.238e1f29.000000000000__head_ae14d5be__8

It is visible that Ceph uses COW (copy on write) to implement snapshot: Before writing to object, copy it as the data object of snapshot, and then continue to modify it in object.

(5) Execute command DD if=/dev/sda1 of=/dev/rdb1 bs=1048576 seek=4 count=4 writes [4MB,8MB] data to image. The operation creates a second data object. Because this is created after doing snapshot, all of it has nothing to do with snapshot.

(6) Create another snapshot, then modify the second data object, this time the second data object in the same folder as the snapshot of a data object file:

[Email protected]:/data/osd2/current/8.2c_head# ls/data/osd/current/8.a_head/-ltotal 4100-rw-r--r--1 root root 4194304 Sep 03:31 rb.0. Fc9d.238e1f29.000000000001__head_9c84738a__8[email protected]:/data/osd2/current/8.2c _head# ls/data/osd/current/8.a_head/-ltotal 8200-rw-r--r--1 root root 4194304 Sep 03:35 rb.0.fc9d.238e1f29.00000 0000001__b_9c84738a__8-rw-r--r--1 root root 4194304 Sep 03:35 rb.0.fc9d.238e1f29.000000000001__head_9c84738a__8 

So

(1) Snapshot data Objects is stored in the same directory as the data objects of image.

(2) The granularity of the snapshot is not the entire image, but the data object in Rados.

(3) When snapshot is created, only a small number of bytes of metadata are added to the image's metadata object, and when the image's data Objects is modified (write), the modified objects is copied (copy), as the snapshot D ATA objects. This is the meaning of COW.

4 Cloning (clone)

To create a clone is to copy the state of one of the Snapshot of an image into an image. If Imagea has a snapshot-1,clone is imageb according to Imagea's Snapshot-1 clone. Imageb the state at this time is exactly the same as the Snapshot-1, and has the corresponding ability of the image, the difference is that Imageb is writable at this time.

4.1 Clone users See

From the user's point of view, a clone is exactly the same as the other RBD image. You can do it snapshot, read/write, change the size, etc., in short, from the user's point of view there is no limit. At the same time, the creation is fast because Ceph only allows clone from snapshot, and snapshot is always read-only.

RBD clone pool101/[email protected] image1snap1clon3[email protected]:~# RBD Info IMAGE1SNAP1CLON3RBD image ' Image1snap1clon3 ':        size 102400 MB in 25600 objects        order (4096 KB objects)        Block_name_prefix : Rbd_data.a8f63d1b58ba        format:2 features:layering parent:pool101/[email protected] overlap:102400 MB
           

See 4.2 Clone of the Ceph system

From a system perspective, clone is also using COW technology, and now you can find out in detail by following these steps:

(1) Create a clone (protect snapshot is required before creating). You will find that there are three more objects in RADOS:

[Email protected]:~# RBD clone Pool101/[email protected] Pool101/image1snap1clone1
[Email protected]:~# RBD ls-p pool101
Image1
Image1snap1clone1
[Email protected]:~# rados-p pool101 ls
Rbd_header.89903d1b58ba
Rbd_directory
Rbd_id.image1snap1clone1
Rbd_id.image1
Rbd_children
rbd_header.a9532ae8944a
rbd_data.a9532ae8944a.0000000000000000

Among them, Rbd_children records the parent-child relationship:

[Email protected]:~# rados-p pool101 listomapvals Rbd_children
Key: (bytes):
0000:08 xx (xx) xx, 0c, XX, and the most of the .... a953
0010:32-----------0e.

Value: (bytes):
0000:01 xx 0c 00 00 00 38 39 39 30 33 64 31 62: ... 89903d1b
0010:35 61:58ba

It's just more than RBD_HEADER.A9532AE8944A,RBD_HEADER.89903D1B58BA. Partent Information:

Parent
Value: (bytes):
0000:01 to the xx, XX (xx) 0c 00. (.............
0010:00 00--------- a9532ae8944a.
0020:00 00 00 00 00 00 00 00 00 00 19 00 00 00: .....

Here are the sources of parent-child relationships and RBD children results:

[Email protected]:~# RBD Children pool101/[email protected]pool101/image1snap1clone1

As can be seen, Clone is also used for snapshot using COW method.

(2) reading data from clone

Read data in the RBD image, which is essentially clone, is read from its parent snapshot for not its own data objects,ceph, and if it does not, continue looking for its parent image until a data object Exist. It can be seen from this process that the process is inefficient.

(3) Write data to object in clone

Ceph first checks to see if the data object on the clone image exists. If it does not exist, the data object is copied from the parent snapshot or image and then the write operation is performed. At this point, clone has its own data object.

[Email protected]:/data/osd3/current/8.32_head# ls-ltotal 4100-rw-r--r--1 root root 4194304 Sep 05:14 rbd\udata.8 9903d1b58ba.0000000000000000__head_ceddc1b2__8

This is the relationship between the objects after the clone has been increased:

4.3 Flatten clone

From the above analysis we know that the cloning operation essentially replicates a metadata object, and the data objects does not exist. Therefore, each read operation is first accessed to the possible data object of this volume. Accessing the corresponding object to the parent volume after the returned object does not have an error ultimately determines whether the data exists. Therefore, when there are multiple levels of clone chains, read operations require more loss to read the data objects of the superior volume. You do not need to access the parent volume until the data object of this volume is present (that is, after the write operation).

To prevent too many parent-child layers, Ceph provides the Flattern function to copy the data objects shared by clone with parent snapshot to clone, and to delete the parent-child relationship.

Flatten methods for RBD tools:

RBD flatten <image-name>: Fill Clone (Image-name) with data of the parent (make it independent)
If an image is a clone, copy all shared blocks (data Objects) from its parent snapshot, deleting the dependency on its parent. At this point, its parent snapshot can be protected (unprotected), and if there is no other clone, it is allowed to be deleted. This feature requires that image be formatted as Format 2.

Note that this is a very time-consuming operation. After Flatten, clone and the original parent snapshot no longer have a relationship, really become a separate image:

[Email protected]:~# RBD Flatten image1snap1clon2image flatten:100% complete...done.[ Email protected]:~# RBD Info image1snap1clon2rbd image ' Image1snap1clon2 ':        size 102400 MB in 25600 object s        order (4096 KB objects) block_name_prefix:rbd_data.fb173d1b58ba format:2 features:layering  

Surprisingly, this operation produced a large number of empty files in the directory of the image that was produced by the parent image and clone:

[Email protected]:/data/osd/current# ls/data/osd/current/8.7f_head/-ltotal 788-rw-r--r--1 root root 0 Sep 28 05:33< c1> rbd\udata.89903d1b58ba.000000000000014f__head_17c7607f__8-rw-r--r--1 root root 0 Sep 05:33 rbd\ udata.89903d1b58ba.0000000000000232__head_96d143ff__8-rw-r--r--1 root root 0 Sep 05:33 rbd\ udata.89903d1b58ba.0000000000000399__head_4d4e557f__8-rw-r--r--1 root root 0 Sep 05:33 rbd\ udata.89903d1b58ba.00000000000003ae__head_ce165dff__8-rw-r--r--1 root root 0 Sep 05:33 rbd\ udata.89903d1b58ba.00000000000003e1__head_42ea8a7f__8-rw-r--r--1 root root 0 Sep 05:33 rbd\ Udata.89903d1b58ba.0000000000000445__head_701607ff__8     

Determine the number of parent-child relationship layer, reach a certain number after flatten clone, delete snapshot pseudo-code:

img = Self.rbd.Image (client.ioctx, img_name) #根据输入的 img_name get its RBD Image object
_pool, parent, snap = Self._get_clone_info (img_name) #当 the RBD image with name Img_name is a clone, gets its parent image by recursion and parent Snapshotimg.flatten () # Copies the data of the parent snapshot into the clone Parent_volume = self.rbd.Image (client.ioctx, par ENT) #获取 The parent imageparent_volume.unprotect_snap (SNAP) #将 Snap to protect Parent_volume.remove_snap (SNAP) #如果snapshot no other Clone, delete it 

Reference Links:

http://www.wzxue.com/ceph-librbd-block-library/

Http://docs.ceph.com/docs/master/architecture/

https://www.ustack.com/blog/ceph_infra/

https://hustcat.github.io/rbd-image-internal-in-ceph/

http://www.wzxue.com/category/ceph-2/

Understanding OpenStack & Ceph (4): Ceph's underlying data structure [Pool, Image, Snapshot, Clone]