How to not start the Container Mount container file system

Content Introduction

In Docker practice, there are sometimes some stability issues, such as the inability of Docker containers to start, especially in earlier versions of Docker. In addition, in the container image is sometimes due to improper configuration of the container can not start up, this time, if you understand the principle of the storage technology used by Docker, then it is possible to implement the container's data volume to mount to the host's specified directory, and then to modify or back up the data work To restore the container to normal and restore important data

Technical principle

1. The typical structure of the Docker image is shown. The traditional Linux load Bootfs will first set the Rootfs to Read-only, then change the rootfs from Read-only to Read-write after the system self-test, then we can write and read on the Rootfs. However, the image of Docker is not so, it will not change the Rootfs read-only to Read-write after Bootfs self-test. Instead, the rootfs of one or more read-only is loaded onto the ROOTFS layer of the previous read-only using the Union mount (a mount mechanism of UnionFS). After loading so many layers of rootfs, it still makes it look like a filesystem, and in the Docker system, these read-only rootfs of the union mount are called Docker mirrors. However, at this point each layer of ROOTFS is read-only, and we cannot operate on it at this time. When we create a container, that is, the Docker image is instantiated, the system allocates a layer of empty read-write rootfs on top of a layer or multilayer read-only rootfs.

Docker mirroring this hierarchical organization has many benefits, first saving the space occupied by the mirror on the physical machine, and, secondly, creating a new, empty rootfs, which means that containers can be started faster than other virtualization technologies.

2. The Docker community recommends using AUFS to organize its hierarchical mirror structure, but on the CentOS platform, because Aufs is not included in the kernel, Docker uses devicemapper as the underlying storage technology for mirroring and containers. Devicemapper is a generic device mapping mechanism in the Linux kernel that supports logical volume management, and simply, through the interface provided by Devicemapper, you can create logical block devices and corresponding mapping tables that can be mounted to Docker containers for use. When the IO request of the container falls to the specified block device, the Devicemapper kernel module locates the IO request to the specified physical block device based on the previously created mapping table and completes the IO operation. Each layer of a docker image corresponds to a logical block device in the Devicemapper.

3, Devicemapper provides a variety of mapping management scheme, which has a snapshot mechanism called thin-provisioning, specific details can be referred to reference 3, in short, is devicemapper to meet the requirements of Docker to mirror layered organization, That is, each layer of mirroring corresponds to a DM device, and these DM devices can be mount into a file system for the container to use, and the DM device on the level overlay is no depth limit

4. Below, from the Docker code level, analyze how Docker manages the Docker image through the mechanism provided by Devicemapper, and the main code is in the Docker source directory pkg/devicemapper/ In the Devicemapper.go file:
A, when the Docker daemon is initialized, a thin pool is created first through the interface provided by Devicemapper, and each subsequent logical device is allocated from this pool:

func CreatePool(poolName string, dataFile, metadataFile *os.File, poolBlockSize uint32) error {    ......    params := fmt.Sprintf("%s %s %d 32768 1 skip_block_zeroing", metadataFile.Name(), dataFile.Name(), poolBlockSize)    if err := task.AddTarget(0, size/512, "thin-pool", params); err != nil {        return fmt.Errorf("Can‘t add target %s", err)    }    ......    if err := task.Run(); err != nil {        return fmt.Errorf("Error running DeviceCreate (CreatePool) %s", err)    }    return nil}

The above code is equivalent to executing a command similar to this in user space:

dmsetup create docker-253:0-756230-pool --table "0 209715200 thin-pool /dev/sdb1 /dev/sdb2 128 32768 1 skip_block_zeroing"

A pool with a specified number of sectors is created, and then a logical device named Dm-x can be created from the pool
B. Create a basic thin device from the pool with the following code:

func CreateDevice(poolName string, deviceId int) error {    ......    if err := task.SetMessage(fmt.Sprintf("create_thin %d", deviceId)); err != nil {        return fmt.Errorf("Can‘t set message %s", err)    }    ......}

The above code is equivalent to executing the following command in user space:

dmsetup message docker-253:0-756230-pool 0 "create_thin 1"

A baseimage (ID of 1) was created, and all subsequent mirroring layers were based on the snapshot formed by this baseimage
C, after each mirror layer created, the action to create the snapshot is performed:

func CreateSnapDevice(poolName string, deviceId int, baseName string, baseDeviceId int) error {    ......    if err := task.SetMessage(fmt.Sprintf("create_snap %d %d", deviceId, baseDeviceId)); err != nil {        if doSuspend {            ResumeDevice(baseName)        }        return fmt.Errorf("Can‘t set message %s", err)    }    ......}

The above code is equivalent to executing the following command in user space:

dmsetup message docker-253:0-756230-pool 0 "create_snap 2 1"

This will register a new device (ID 2) in the kernel space, which is based on the device created in the previous step.
D, then activate the snapshot device, you can see the device in the user space, and then mount it to the container to use:

func ActivateDevice(poolName string, name string, deviceId int, size uint64) error {    ......    params := fmt.Sprintf("%s %d", poolName, deviceId)    if err := task.AddTarget(0, size/512, "thin", params); err != nil {        return fmt.Errorf("Can‘t add target %s", err)    }    ......}

The above code is equivalent to executing in user space:

dmsetup create dm-1 --table "0 20971520 thin /dev/mapper/docker-253:0-756230-pool 2"

After execution, the/DEV/DM-1 device is seen in the system, and Docker can then mount the device to the specified container for use. When the container is closed, the logical device of dm-1 is deleted, but in fact the device with ID 2 registered in the kernel still exists and is not actually deleted.

Specific steps

Having learned how Docker manages the image hierarchy through Devicemapper, we can mount the file system of a container that cannot be started by using the following method, which is actually mapping the device with the ID number of the container that is not deleted in the kernel:
1. Docker ps-a Query the ID number of the virtual machine to be mounted

    CONTAINER ID        IMAGE                      COMMAND                CREATED             STATUS              PORTS               NAMES1bd4cc1aed92        mgj-base-20150807:latest   "/usr/bin/python2.6    5 hours ago         Up 5 hours                              10.13.130.56bcee5c0e31ea        mgj-base-20150807:latest   "/usr/bin/python2.6    5 hours ago         Up 5 hours                              10.13.130.57

2. Obtain the device number information for the container, for example:

cat /var/lib/docker/devicemapper/metadata/1bd4cc1aed92fae13cdf3ee586460a0319d9ea4b95c039b2ec0325a9ff9ff437

Get the following content:

{"device_id":6,"size":107374182400,"transaction_id":111,"initialized":false}

The device number in Devicemapper is 6, and the 4th step behind it is
3. Perform dmsetup ls to view Docker pool information:

docker-8:1-44303307-pool (253:0)

The 4th step is to use the above query results
4, the 2nd step, the size obtained is 107374182400 bytes, and the Dmsetup configuration is calculated as the number of sectors, so size should be 107374182400/512 = 209715200 sectors, execute the following command:

dmsetup create tmp --table "0 209715200 thin /dev/mapper/docker-8:1-44303307-pool 6"

A temporary DM device named TMP was created and the device_id mapped to is 6, which is the device number of the execution container
A directory named TMP is now available in the/dev/mapper/directory
5. Implementation

mount /dev/mapper/tmp /home/tmp

Enter the/home/tmp/rootfs directory to see the data inside the specified container, can be modified or can do data backup, etc.

How to not start the Container Mount container file system