docker深入2-儲存驅動之使用devicemapper(direct-lvm)模式

標籤：system   display   minimum   b/s   nim   monitor   準備   hold   conf   

docker深入2-儲存驅動之使用devicemapper(direct-lvm)模式

2016/12/9

一、現狀1、預設使用的是 devicemapper(loop-lvm)[[email protected] lib]# docker info     （略）        Server Version: 1.12.3Storage Driver: devicemapper Pool Name: docker-253:3-33517-pool Pool Blocksize: 65.54 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file: /dev/loop0 Metadata file: /dev/loop1 Data Space Used: 11.8 MB Data Space Total: 107.4 GB Data Space Available: 7.433 GB Metadata Space Used: 581.6 kB Metadata Space Total: 2.147 GB Metadata Space Available: 2.147 GB Thin Pool Minimum Free Space: 10.74 GB Udev Sync Supported: true Deferred Removal Enabled: false Deferred Deletion Enabled: false Deferred Deleted Device Count: 0 Data loop file: /var/lib/docker/devicemapper/devicemapper/data WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device. Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata（略）Docker Root Dir: /var/lib/docker（略）2、對應的裝置[[email protected] lib]# lsblk -I 7NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTloop0                       7:0    0  100G  0 loop └─docker-253:3-33517-pool 252:0    0  100G  0 dm   loop1                       7:1    0    2G  0 loop └─docker-253:3-33517-pool 252:0    0  100G  0 dm 二、調整為 devicemapper(direct-lvm)1、準備工作安裝 lvm2[[email protected] ~]# yum install lvm2增加一塊磁碟[[email protected] ~]# ls /dev/vdb/dev/vdb停止 docker 服務[[email protected] ~]# systemctl stop docker2、配置裝置1）pv[[email protected] ~]# pvcreate /dev/vdb2）建立一個名稱為 docker 的 vg [[email protected] ~]# vgcreate docker /dev/vdb3）建立2個 lv，分別對應 data 和 metadata 裝置[[email protected] ~]# lvcreate --wipesignatures y -n thinpool docker -l 95%VG[[email protected] ~]# lvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VG4）轉換 pool 為 thinpool 格式[[email protected] ~]# lvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmeta5）調整 lvm 配置The value should be the percentage of space used before lvm attempts to autoextend the available space (100 = disabled).定義一個百分比的閾值，表明觸發 lvm 自動擴容前，已用空間佔比。thin_pool_autoextend_threshold = 80The value’s setting is the percentage of space to increase the thin pool (100 = disabled)每次擴容 thin pool 空間的比例thin_pool_autoextend_percent = 20[[email protected] ~]# cat /etc/lvm/profile/docker-thinpool.profileactivation {    thin_pool_autoextend_threshold=80    thin_pool_autoextend_percent=20}6）lvchange 應用配置[[email protected] ~]# lvchange --metadataprofile docker-thinpool docker/thinpool7）lvs 查看卷的資訊，驗證 monitor 的狀態[[email protected] ~]# lvs -o+seg_monitor  LV       VG     Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Monitor    thinpool docker twi-a-t--- 190.00g             0.00   0.01                             monitored3、調整 docker 儲存1）備份[[email protected] ~]# mkdir /var/lib/docker.bk[[email protected] ~]# mv /var/lib/docker/* /var/lib/docker.bk如果這個 docker engine 上已經有部分 images 在使用，且需要儲存，請先提前推送到自己的 registry 中儲存。2）調整服務配置a、第一種方式[[email protected] ~]# vim /lib/systemd/system/docker.service 增加參數：--storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true或者：[[email protected] ~]# sed -i ‘/^ExecStart=/c\ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true‘ /lib/systemd/system/docker.service b、第2種方式調整 daemon.json 的配置：[[email protected] ~]# cat /etc/docker/daemon.json{    "storage-driver": "devicemapper",    "storage-opts": [        "dm.thinpooldev=/dev/mapper/docker-thinpool",        "dm.use_deferred_removal=true",        "dm.use_deferred_deletion=true"    ]}c、注意事項Note: Always set both dm.use_deferred_removal=true and dm.use_deferred_deletion=true to prevent unintentionally leaking mount points.啟用上述2個參數來阻止可能意外產生的掛載點泄漏問題3）重啟服務[[email protected] ~]# systemctl daemon-reload[[email protected] ~]# systemctl start docker4）驗證[[email protected] lib]# docker info     （略）  Storage Driver: devicemapper Pool Name: docker-thinpool Pool Blocksize: 524.3 kB Base Device Size: 10.74 GB Backing Filesystem: xfs Data file:  Metadata file:  Data Space Used: 20.45 MB Data Space Total: 204 GB Data Space Available: 204 GB Metadata Space Used: 266.2 kB Metadata Space Total: 2.143 GB Metadata Space Available: 2.143 GB Thin Pool Minimum Free Space: 20.4 GB Udev Sync Supported: true Deferred Removal Enabled: true Deferred Deletion Enabled: true Deferred Deleted Device Count: 0 Library Version: 1.02.107-RHEL7 (2016-06-09)（略） 5）可以通過 lvs, lvdisplay, vgs 等指令來查看 lvm 卷的狀態6）可以查看日誌，瞭解 thin pool 在自動擴容觸及閾值時的狀態[[email protected] ~]# journalctl -fu dm-event.service7）對應的裝置[[email protected] ~]# lsblk -I ‘253‘ |grep -v vdaNAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTvdb                     253:16   0  200G  0 disk ├─docker-thinpool_tmeta 252:0    0    2G  0 lvm  │ └─docker-thinpool     252:2    0  190G  0 lvm  └─docker-thinpool_tdata 252:1    0  190G  0 lvm    └─docker-thinpool     252:2    0  190G  0 lvm 8）確認無誤後，可以清理舊的資料[[email protected] ~]# rm /var/lib/docker.bk -fr4、整合到一個小指令碼中來配置direct-lvm的儲存[[email protected] ~]# cat docker_direct_lvm_setup.sh    #!/bin/bash##2016/12/8#v1.0.5 @PC## 分配一塊獨立的磁碟供docker使用，本例使用的是 /dev/vdbdev_name=‘/dev/vdb‘yum -y -q install lvm2echo ‘+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++‘pvcreate ${dev_name}vgcreate docker ${dev_name}lvcreate --wipesignatures y -n thinpool docker -l 95%VGlvcreate --wipesignatures y -n thinpoolmeta docker -l 1%VGlvconvert -y --zero n -c 512K --thinpool docker/thinpool --poolmetadata docker/thinpoolmetacat <<‘_EOF‘ >/etc/lvm/profile/docker-thinpool.profileactivation {    thin_pool_autoextend_threshold=80    thin_pool_autoextend_percent=20}_EOFlvchange --metadataprofile docker-thinpool docker/thinpoolecho ‘+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++‘lvs -o+seg_monitorecho ‘+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++‘cat <<‘_EOF‘+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++接下來的操作步驟樣本：[1] 如果 docker 有舊的資料，先推送到registry中再移除；# mkdir /var/lib/docker.bk# mv /var/lib/docker/* /var/lib/docker.bk[2] 更新docker服務的配置：(方式一： 調整 docker.service 的配置參數)# sed -i ‘/^ExecStart=/c\ExecStart=/usr/bin/dockerd --storage-driver=devicemapper --storage-opt=dm.thinpooldev=/dev/mapper/docker-thinpool --storage-opt=dm.use_deferred_removal=true --storage-opt=dm.use_deferred_deletion=true‘ /lib/systemd/system/docker.service (方式二： 在 daemon.json 中配置參數)# cat /etc/docker/daemon.json{    "storage-driver": "devicemapper",    "storage-opts": [        "dm.thinpooldev=/dev/mapper/docker-thinpool",        "dm.use_deferred_removal=true",        "dm.use_deferred_deletion=true"    ]}[3] 重啟服務# systemctl daemon-reload && systemctl start docker[4] 驗證是否調整完畢# docker info+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++_EOFZYXW、參考1、docshttps://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#/configure-direct-lvm-mode-for-production2、Docker Device Mapper 使用 direct-lvmhttp://www.cnblogs.com/SZLLQ2000/p/5486834.html3、daemon.jsonhttps://docs.docker.com/engine/reference/commandline/dockerd/

docker深入2-儲存驅動之使用devicemapper(direct-lvm)模式