Quickly distribute Docker images by peer

When deploying a large container application cluster, it often takes a lot of time to publish the application image to all nodes. Our VMware Research and development team has tested the approach to the peer to be able to better solve the problem of large-scale image distribution, providing good guidance for operation and maintenance practice.

Overview

When using Docker to run containerized apps, the host usually first downloads the corresponding image (image) from the registry service (such as the Docker Hub). This mirroring mechanism is useful in a development environment, and it is easy for team members to share the same image. In the actual production environment, from the point of view of efficiency and security, the private registry service is often deployed, which is used for the production line machine cluster. When a large number of hosts need to simultaneously run the container application from the Registry download image (such as the release of new version, hit the nail, etc.), the Registry service will often become the bottleneck of the mirror distribution, the application image will take a long time to transfer to all hosts, so that the application release cycle greatly extended. Many users take a pre-download image method, let each node in advance to obtain the image in batches, and then at a predetermined time to launch the application uniformly. This approach eased the problem to some extent, but in essence did not solve the bottleneck of registry distribution.

To solve this problem, we set out to implement a system that uses the bit torrent protocol to speed up the image distribution (decentralized image distribution,did), whose main idea is to allow mirroring to be shared between different hosts. Create a distributed peer-to download network to improve network throughput. Through testing, we have verified that the system has a large performance advantage in the case of large number of cluster nodes and larger image.

Architecture

Figure 1 Did system architecture

Administration Interface Admin Console
The admin console allows the administrator to define the task under which the image is issued (such as the size of the cluster, the IP address of the machine, the image URL, etc.) and the completion of the task in real time.

Controller
The controller is the core component of the did system and controls the entire process of image distribution. After receiving the image distribution task from the admin console, the controller completes the mirroring preparation and assigns the specific image download task to the client agents of each node.

Client Agent
The client agent is deployed in each node of the cluster, cooperating with the scheduling of the controller to complete the entire image distribution process. The agent receives the mirror download task from the controller, invokes the BT client to download the image, and eventually imports the image into the Docker daemon.

BT Client
The BT client deployed on the cluster node and the BT client deployed in the controller, together with the tracker, form a complete peer to file transfer system. During the distribution of the entire image, they use the BT protocol to complete the image download.

BT Tracker
Tracker is part of the BT system, which stores the metadata information required by the BT client during the download process and assists each BT client in completing the entire communication process.

principle of mirror distribution
When a user submits a mirror distribution task (job) to the did system through the admin console, the controller handles the following:

? Download the image from registry to the local mirror repository via the local Docker Daemon REST API,  
call Docker Daemon API Export Image file (image tar file) from the mirrored warehouse;  

The following processing occurs after the client agent receives a mirror download task from the controller:

? For each layer of the image, by calling to check if it exists locally, add the nonexistent layer ID to the list to be downloaded;
? Download the corresponding seed file to the layer in the download list, and start the BT client to complete the download of the layer file;
? Import the completed layer file into the Docker daemon via the Docker Daemon API;
? Repeat steps 2 and 3 until all the layers you want to download are imported into the Docker daemon. Note: The download process for each layer is performed concurrently.

Because Docker mirrors are stored in layers, different mirrors can share layers, which not only reduces the consumption of storage, but also downloads the missing layer when downloading the image, thus reducing the download time of the image. In the above did image distribution process, we split the image into layers that utilize the BT protocol for transmission. In addition, since the BT protocol was originally designed for file distribution on the Internet, some of the designs did not apply to scenarios where local area network images were distributed centrally. For example, in the BT download process, each client maintains a peer list that participates in downloading the file, and by default the BT client will communicate with tracker once every 1800 seconds to update. This time was modified in the did design in order to 1 seconds so that the BT client of each node could get the information of the other peer more quickly.

Experimental environment and results

Due to the limitations of the test environment, we have built an experimental environment consisting of 10 physical machines and 10 virtual machines on each physical machine to simulate a cluster scenario with 100 nodes. 10 physical machines are connected to the 1Gbps Ethernet switch. Each virtual machine is configured with 2 Vcpus, 4GB of memory, and a 500GB hard drive.

The first set of experiments was used to compare the effects of different image sizes on the download process in clusters with a certain number of nodes (such as 100 nodes). As shown in experiment result 2, the propagation time of the vertical axis in the figure is the period from which the entire cluster begins to download the image to all nodes that will mirror the import:

Figure 2 Docker, did mirror distribution time comparison

The two-point rule can be seen from the test results. The first is that the download time for Docker and did increases as the mirror grows, but did has a smaller growth slope. And as the image increases, the benefits of did become more apparent, for example, when the mirror exceeds 500MB, did does not take half the time required for Docker. Second, there is such a balance point, when the image is less than this value, Docker download faster, because the direct download of Docker at this time did not cause too much network congestion. In addition, in the process of downloading with BT protocol, the BT client and tracker need to communicate with each other and the client, and the flow of control can be consumed in some degree. , the additional overhead of the BT protocol can be compensated by network savings, resulting in overall performance gains when mirroring increases above 130MB.

Since both approaches have pros and cons, we did a second set of experiments to study the balance between Docker and did performance. When Docker and did distribute the same image in the same cluster in equal time, the number of cluster nodes and the mirror size are called a balance point. The results of Experiment 3 show:

Figure 3 The balance point of Docker and did

It can be seen from the experimental results that as the number of cluster nodes increases, the image size required to reach the equilibrium point is decreasing. That is to say, the performance advantage is most obvious in the scene with large number of nodes and large mirror.

We have designed and implemented the did image publishing mechanism, using the BT protocol to complete the image distribution process, compared to Docker's own download mode, in the case of heavy load has a more obvious advantage. The native BT protocol is not fully adapted to the usage scenarios of image distribution, and did have been partially optimized for it. However, we believe there are many places that can be adjusted, and the performance should be improved to a certain extent after optimization.

The transfer of mirror-to-peer is a hot topic, and there are few test data in this area. Our test results can be used as a basis for the development of similar systems in the future. For example, in the Enterprise-class registry Project Harbor, where our team has open source, not only does it provide complete user Rights control (RBAC) and a friendly user management interface, it also plans to use peer-to-peer transmission to enhance the efficiency of image publishing and provide strong support for rapid release of container applications in production environments. You are welcome to pay attention to and use Harbor Open source Registry project:

Https://github.com/vmware/harbor

Quickly distribute Docker images by peer