The three---the "Write your own Docker" book, Introduction to the Union File system

Union File System UnionFS

UnionFS is a file system service designed for the LINUX,FREEBSD and NetBSD operating systems to federate other file systems to a single federated mount point. It uses branch to "transparently" Overwrite files and directories of different file systems to form a single, consistent file system. These branches are either read-only or read-write, so when you write to the virtual federated file system, the system is actually written into a new file. It seems that this virtual federated file system can operate on any file, but in fact it does not change the original file, because UnionFS used an important management technology called write-time replication.
Copy-on-write (Copy-on-write, hereinafter referred to as Cow), also known as implicit sharing, is a resource management technique that enables efficient replication of modifiable resources. The idea is that if a resource is duplicated, but there is no modification, it is not necessary to create a new resource immediately; The resource can be shared by the old and new instances. Creating a new resource occurs during the first write, that is, when the resource is modified. This sharing of resources can significantly reduce the consumption of unmodified resource replication, but will also increase the cost of small portions when making resource modifications.
In a classic example, Knoppix, a Linux distribution for Linux demos, CD-ROM tutorials, and commercial product demos, is to put a CD-ROM or DVD and one that exists in a writable device (eg, USB flash drive) on the file system called Knoppix.img together. Any changes to the files on the CD/DVD will be applied to the USB drive without changing the contents on the original CD/DVD. AUFS

AUFS, English full name is advanced multi-layered unification filesystem, once also called acronym multi-layered Unification Filesystem,another Multi-layered unification filesystem. Aufs completely rewritten the early UnionFS 1.x, its main purpose is for reliability and performance, and introduces some new features such as load balancing of writable branches. Some implementations of AUFS have been incorporated into the UnionFS 2.x version. How Docker uses Aufs's

Aufs is the first storage driver for Docker. Aufs has the advantage of fast boot containers, efficient use of storage and memory, until now AUFS is still a type of storage driver supported by Docker. Next we'll show you how Docker uses AUFS to store images and containers. image layer and Aufs

Each Docker image is made up of a series of read-only layers. The contents of image layers are stored in the/var/lib/docker/aufs/diff directory of the Docker hosts filesystem. The/var/lib/docker/aufs/layers directory, however, stores how the image layer stacks these layer metadata.
prepare an ECS with Docker 1.11.2 installed. Without pulling any mirrors, launching any containers, execute the Ls/var/lib/docker/aufs/diff command and discover that the directory does not store anything. Pull the ubuntu:15.04 mirror, and then execute the ls/var/lib/docker/aufs/diff command again. We can see that the results of the Docker pull show a total of 4 layers in the ubuntu:15.04 image, and there are four corresponding storage file directories in the results of the Execute Ls/var/lib/docker/aufs/diff command. One thing to note here is that since Docker 1.10, the storage Mirror Layer folder in the diff directory is no longer the same as the mirror ID. Last cat/var/lib/docker/aufs/layers/ The 6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea command lists the 6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205 in the stack. Ec165d2ad8a44e148ea the layers below the layer.

$ docker Pull ubuntu:15.04 15.04:pulling from Library/ubuntu 9502adfba7f1:pull complete 4332ffb06e4b:pull complete 2f93 7cc07b5f:pull Complete A3ed95caeb02:pull Complete digest:sha256 : 2fb27e433b3ecccea2a14e794875b086711f5d49953ef173d8a03e8707f1510f status:downloaded Newer image for ubuntu:15.04 $ ls  /var/lib/docker/aufs/diff 208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573  6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77 $ ls/var/lib/docker/aufs/mnt  208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573  6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77 $ cat/var/lib/docker/aufs/layers/ 6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea 9C444E426A4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77
 208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b

Next we will create a mirror named Changed-ubuntu with the ubuntu:15.04 image as the base image. This image simply adds a "Hello world" file to the mirrored/TMP folder. You can use the following dockerfile to implement:

From ubuntu:15.04

 RUN Echo ' Hello World ' >/tmp/newfile

In the terminal CD to the upper Dockerfile location, perform the Docker build-t changed-ubuntu. command to build the image.

$docker build-t Changed-ubuntu.
Sending build context to Docker daemon 10.75 kB
step 1:from ubuntu:15.04
 ---> d1b55fd07600
step 2:run E Cho "Hello World" >/tmp/newfile
 ---> Running in c72100f81dd1
 ---> 9d8602c9aee1
removing Intermediate container c72100f81dd1
successfully built 9d8602c9aee1

Then perform Docker images to view the current image and see the newly generated changed-ubuntu.

$docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
changed-ubuntu      latest              9d8602c9aee1 about        a minute ago   131.3 MB
ubuntu              15.04               d1b55fd07600 months        ago        131.3 MB

Using the Docker History changed-ubuntu command, you can clearly see which image layers the Changed-ubuntu image uses. From the output you can see that the 9D8602C9AEE1 image layer is at the top level and only 12B in size, created by the/bin/sh-c echo "Hello World" >/tmp/newfile command. This means that the Changed-ubuntu image takes up only 12Bytes of disk space, which proves how efficiently aufs uses disk space. The following four-layer image layers is a shared 4 image layers that make up the ubuntu:15.04 image. The layers of the "missing" tag is from Docker 1.10, and the image history data for a mirrored image layers is stored in a file, which is a normal behavior that Docker officially deems.

 $docker history Changed-ubuntu IMAGE CREATED CREATED by SIZE COMMENT 9d8602c9aee1 4 minutes ago/bin/sh-c echo "Hello World" &G T /tmp/newfile b d1b55fd07600 months ago/bin/sh-c # (NOP) CMD ["/bin/bash"] 0 B <missi  Ng> months ago/bin/sh-c sed-i ' s/^#\s*\ (deb.*universe\) $/1.879 KB <missing> 10 Months ago/bin/sh-c echo ' #!/bin/sh ' >/usr/sbin/polic 701 B <missing> months ago/ Bin/sh-c # (NOP) ADD file:3f4708cf445dc1b537 131.3 MB 

Next we continue to look at the storage information of layers, from the output we can see the/var/lib/docker/aufs/diff directory and/var/lib/docker/aufs/ The MNT directory is one more folder 9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e. When using cat/var/lib/docker/aufs/layers/ 9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e command to view its metadata when you can see that the layers in front of it is Ubuntu : The 4 image layers used by the 15.04 image. Further probing the/var/lib/docker/aufs/diff/9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e folder and discovering that it stores A/ Tmp/newfile file, there is only one line in the file "Hello World". At this point, we have completely analyzed how image layer and Aufs are shared between files and folders for mirrored storage.

$ Ls/var/lib/docker/aufs/diff 208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b  9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77  6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea
9F122DBAA103338F27BAC146326AF38A2BCB52F98EBB3530CAC828573FAA3C4E $ ls/var/lib/docker/aufs/mnt  208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b  9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77  6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea 9F122DBAA103338F27BAC146326AF38A2BCB52F98EBB3530CAC828573FAA3C4E $ cat/var/lib/docker/aufs/layers/
9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e
6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573 F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73A98119e2cf77 208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b $ cat/var/lib/docker/aufs/diff/ 9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e/tmp/newfile Hello World

container layer and Aufs

Docker uses Aufs's cow technology to achieve image layer sharing and reduce disk space consumption. Cow means that once only a small part of a file is changed, Aufs also needs to copy the entire file. This design can have a certain effect on the performance of the container, especially if the file to be copied is large, or is located under many image layers, or aufs requires a deep search of the directory tree. But don't worry too much, for a container, each image layer needs to be copied at most once. Subsequent changes will be made on the container layer of the first copy.
When you start a container, Docker creates a read-only init layer for it to store content related to the environment within the container, and Docker creates a read-write layer for it to perform all the writes.
The mount directory of the Container layer is also/var/lib/docker/aufs/mnt. The container metadata and configuration files are stored in the/var/lib/docker/containers/directory. The container read-write layer is stored in the/var/lib/docker/aufs/diff/directory. Even if the container is stopped, the read-write layer still exists, so restarting the container does not lose data, and the read-write layer is deleted only when a container is deleted.
Next we still use experiments to prove the above conclusions. The number of existing containers is first queried at 0, and no data is found in the/var/lib/docker/containers directory. Finally, look at the Aufs Mount case of the system, with only one config file.

$ docker ps-a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

$ ls/var/lib/ Docker/containers

$ ls/sys/fs/aufs/

Launches a Changed-ubuntu container.

$docker run-dit changed-ubuntu Bash
fb5939d878bb0521008d63eb06adea75e6af275855f11879dfa3992dfdaa5e3f

$ Docker ps-a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
FB5939D878BB        changed-ubuntu      "bash"              seconds ago up      seconds                           amazing_babbage

View/var/lib/docker/aufs/diff directory discovery, two more folders below, F9CCF5CAA9B7324F0EF112750CAA14203B557D276CA08C78C23A42A949E2BFC8 is the read-write layer that Docker creates for the container, The f9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init is the init layer of the read-only that Docker created for the container.

$ Ls/var/lib/docker/aufs/diff
208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b  9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e  F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init
6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea  F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573  F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8

/VAR/LIB/DOCKER/AUFS/MNT directory changes are consistent with/var/lib/docker/aufs/diff.

$ ls/var/lib/docker/aufs/mnt
208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b  9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e  F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init
6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea  F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573  F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8

The/var/lib/docker/aufs/layers/directory has more files with the same name as the above two files, and the Cat command allows you to clearly see its dependent layer records.

$ls/var/lib/docker/aufs/layers 208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b  9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e
F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init  6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77  9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573 F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8 $ cat/var/lib/docker/aufs/layers/
F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8
F9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init
9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e
6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77 208319B22189A2C3841BC4A4EF0DF9F9238A3E832DC403133FB8AD4A6C22B01B $ cat/vAr/lib/docker/aufs/layers/f9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init
9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e
6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea
9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573
F193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77 208319b22189a2c3841bc4a4ef0df9f9238a3e832dc403133fb8ad4a6c22b01b

Create a new folder in the/var/lib/docker/containers/directory that is the same as Containerid, which holds the metadata and config files for the container.

$ ls/var/lib/docker/containers/
fb5939d878bb0521008d63eb06adea75e6af275855f11879dfa3992dfdaa5e3f

$ ls/ var/lib/docker/containers/fb5939d878bb0521008d63eb06adea75e6af275855f11879dfa3992dfdaa5e3f/
Config.v2.json  fb5939d878bb0521008d63eb06adea75e6af275855f11879dfa3992dfdaa5e3f-json.log  Hostconfig.json  Hostname  hosts  resolv.conf  resolv.conf.hash  SHM

Next we look at the situation of mount from the system Aufs, and in the/sys/fs/aufs/directory, we have one more si_fe6d5733e85e4904 folder. With the cat/sys/fs/aufs/si_fe6d5733e85e4904/* command we can clearly see that this is the layer permission of the container we just started, Only the topmost F9CCF5CAA9B7324F0EF112750CAA14203B557D276CA08C78C23A42A949E2BFC8 layer is the Read-write permission.

 $s/sys/fs/aufs/config si_fe6d5733e85e4904 $ cat/sys/fs/aufs/si_fe6d5733e85e4904/*/var/ lib/docker/aufs/diff/f9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8=rw/var/lib/docker/aufs/ diff/f9ccf5caa9b7324f0ef112750caa14203b557d276ca08c78c23a42a949e2bfc8-init=ro+wh/var/lib/docker/aufs/diff/ 9f122dbaa103338f27bac146326af38a2bcb52f98ebb3530cac828573faa3c4e=ro+wh/var/lib/docker/aufs/diff/ 6bb19cb345da470e015ba3f1ca049a1c27d2c57ebc205ec165d2ad8a44e148ea=ro+wh/var/lib/docker/aufs/diff/ 9c444e426a4a0aa3ad8ff162dd7bcd4dcbb2e55bdec268b24666171904c17573=ro+wh/var/lib/docker/aufs/diff/ f193107618deb441376a54901bc9115f30473c1ec792b7fb3e73a98119e2cf77=ro+wh/var/lib/docker/aufs/diff/ 208319B22189A2C3841BC4A4EF0DF9F9238A3E832DC403133FB8AD4A6C22B01B=RO+WH 70/run/shm/aufs.xino 

Finally, mention Aufs how to delete a file for container. If you want to remove File1,aufs, a. wh.file1 file is generated at the Read-write layer of container to hide all read-only layers of file1 files. At this point, we have clearly described and verified how Docker uses AUFS to manage container layers. Write your own aufs

Let's do this ourselves with a simple command to create a Aufs file system and feel how to use aufs and cow for file management.
First create a Aufs folder under your experiment directory, and then create an mnt folder in the Aufs directory to make a mount point. Then in the Aufs directory to create a folder called Container-layer, there is a file named Container-layer.txt, the contents of the file is I am container layer. Similarly, continue to create 4 folders named Image-layer N in the Aufs directory (with a value of N of 1 4) with a folder named Image−layer {n} (the value of n is 1~4) with a name of Image-layer{n}. TXT file with the contents of I am image layer${n}. Use the following command to check the contents of the file:

$ cd/home/qinyujia/aufs

$ ls
container-layer  image-layer1  image-layer2  image-layer3  Image-layer4  mnt

$ cat container-layer.txt
I am container layer

$ cat image-layer1/image-layer1.txt< C11/>i am Image Layer 1

$cat image-layer2/image-layer2.txt
I am Image Layer 2

$ cat Image-layer3/image-lay Er3.txt
I am image Layer 3

$ cat image-layer4/image-layer4.txt
I am Image Layer 4

The file directory to be Federated is ready, and we'll use the Container-layer and 4 folders named Image-layer${n} to mount to the MNT directory you just created using AUFS. In the Mount Aufs command, we do not specify the permissions of the 5 folders to be mounted, the default behavior is that the first directory on the left dirs specified is Read-write permission, followed by read-only permissions.

$ sudo mount-t aufs-o dirs=./container-layer:./image-layer4:./image-layer3:./image-layer2:./image-layer1 none./mnt< c0/>$ Tree mnt
mnt
├──container-layer.txt
├──image-layer1.txt
├──image-layer2.txt
├── Image-layer3.txt
└──image-layer4.txt

We remember that we have read and write access to the file in the System Aufs directory. Here we still use the cat/sys/fs/aufs/si_fe6d5733e85e5904/* command to confirm the permissions of each directory in the new mount's file system. (Note that si_fe6d5733e85e5904 should be the system created for this mnt mount point, not the one mentioned in the introduction of Docker and AUFS) according to the output we can clearly see that Only the Container-layer folder is Read-write and the rest are read-only permissions.

$ cat/sys/fs/aufs/si_fe6d5733e85e5904/*
/home/qinyujia/aufs/container-layer=rw
/home/qinyujia/aufs/ Image-layer4=ro
/home/qinyujia/aufs/image-layer3=ro
/home/qinyujia/aufs/image-layer2=ro
/home/ Qinyujia/aufs/image-layer1=ro
/home/qinyujia/aufs/container-layer/. Aufs.xino

Let's do an interesting operation and add a line of text "write to Mnt's Image-layer1.txt" to the end of the Mnt/image-layer1.txt file. According to the cow technology we introduced above, we can imagine what kind of behavior we will produce.

$ ECHO-E "\nwrite to Mnt ' s Image-layer1.txt" >>./mnt/image-layer4.txt

We used the cat command to view the contents of the Mnt/image-layer4.txt file, and found that the content really changed from "I am Image Layer 4" to
"I am Image Layer 4

Write to Mnt's Image-layer1.txt "because MNT is just a virtual mount point, because we continue to find out exactly where the file was modified.

$ cat./mnt/image-layer4.txt
I am Image Layer 4

write to mnt ' s image-layer1.txt

We looked at the contents of the Image-layer4/image-layer4.txt file and found that it did not change.

$ cat Image-layer4/image-layer4.txt
I am Image Layer 4

Next, when we check the Container-layer folder, we find a file named Image-layer4.txt, the contents of the file is
"I am Image Layer 4

Write to Mnt ' s image-layer1.txt. That is when we try to mnt/ Image-layer4.txt file for write operations, the system first in the MNT directory to find a file named Image-layer4.txt, copy it to the Container-layer directory of the Read-write layer, and then the Container-layer directory of I Mage-layer4.txt file for write operations. To this, we successfully completed a small demo, the implementation of their own Aufs file system.

$ ls container-layer/
container-layer.txt  image-layer4.txt

$cat container-layer/image-layer4.txt
I am Image Layer 4

write to