Dockone WeChat Share (81): Only product database backup and Recovery container Project practice Experience Summary

This is a creation in Article, where the information may have evolved or changed.
"Editor's note" This article shares the experience of the geo-disaster disaster Recovery project of the database Docker, which is developed and tested in the project for the needs of the remote restoration scenario of the user database, and integrates the network, storage, scheduling, monitoring, mirroring and many other modules. After the completion of the implementation, from the technical summary of the selection, development, pit, test and other aspects of experience.

Project background

The database Docker's offsite backup recovers disaster recovery project, develops and tests the requirements for the offsite backup and recovery scenarios of the user database, and integrates the container network, storage, dispatching, monitoring, mirroring and many other modules. At the same time, it develops the tools of monitoring, resource dispatching, logging and puppet automatic push for the daily operation and maintenance of database.

Through the nature of Docker isolation and rapid deployment, it is possible to run multiple database backup/restore instances on a single physical machine, greatly increasing server usage and saving significant cost. Through the research and transformation of Docker itself and related components, we accumulate valuable development experience from common open source products to the internal production environment. Through the daily operations and monitoring of the database that Docker has been running on, it also accumulates valuable Docker operations experience, providing the basis for larger scale promotion containers.

About Container Technology

Through practice, it proves that container technology is easy to use, manageability, and rapid deployment with natural advantages. In terms of resource utilization, containers are deployed on hundreds of physical nodes, providing about 500 + database disaster preparedness instances, increasing the utilization of hardware resources and saving about 400 physical machine procurement costs. These are the tangible benefits of container technology. In terms of resource allocation and isolation, the virtual machine is not lost. The application of CPU, memory, disk IO, network IO limit and so on ensures the rational use of resources, and prevents the excessive consumption of resources of single instance from the mechanism.

Stability is a point of great concern in the use of container technology and a cornerstone. MySQL backup/restore is a CPU intensive + disk IO intensive + network IO intensive business, which is a big test for Docker daemon. For the moment, limiting the number of containers per host (around 5), the cluster has not been running for three months because of the crash phenomenon caused by heavy container load, or trustworthy. The only related problem is Docker daemon hangs dead, the specific phenomenon is Docker info, Docker PS is not responding, Docker volume, Docker images Normal, the following container is working properly. This is an occasional event that cannot be reproduced and needs to be observed later.

The overheads is extremely low (less than 10%) due to the fact that the container is in process mode and reflects almost the same performance as the physical machine. From the data extraction task results, compared with the physical machine, the use of containers on the success rate has no impact, the efficiency is similar. It's also very much in line with the original expectation that there is no difference between running a container or an external service from a physical machine point of view, it is a process, the only difference is that the parent process is not the same.

The above is the container "RUN" benefits, through the unified development process, the application of micro-service, CI/CD and other aspects of the improvement, can further utilize the container "BUILD", "Ship" advantage, container technology also has a huge potential. The drawbacks of container technology are really not obvious. Hard to mention one is the need for a certain amount of learning costs, change the development process and the way, one is the developer of the container technology acceptance degree. This project only takes less than 200 people/day and is really low for a project that uses new technology. At first we were also worried that the adoption of new technologies would lead to difficulties in the development and promotion, and then actually solve the problem by technical problems, and eliminate the majority of users of the use of Docker doubts, but to help the general application of the technology.

We had some discussion about the choice of the Docker daemon version. Now the Docker community is very active, when we used 1.10.3, and now we have two new versions. In the premise of functional satisfaction, stability is the first consideration. Docker has introduced the CNM network model from 1.9.0, and 1.10 is quite mature. CNM is part of what we want to try in this project. Network and volume plug-in function and stability of the upgrade, began to support disk IO read-write speed limit, Device mapper support, and so on, are the reason for the choice of this version. In addition, the introduction of the Docker plugin is a good way to decouple the relationship between Docker and the underlying module, allowing us to focus on the underlying (network, storage) implementation without modifying the Docker daemon itself, while avoiding the need for upgrade dependencies.

About Container Network technology

The container network infrastructure uses the Contiv Netplugin, which is an open source solution from Cisco. Netplugin is plugged into the Docker daemon as a network plug-in, and the network function is called as part of the lifecycle of the container. Netpluign through the management of the OvS, based on the OvS VLAN for isolation, the container allocation of external IP, can be directly accessed, greatly simplifying the way the container access. The reason to consider using this scenario is: 1. Plug-in form does not create an upgrade dependency on Docker. 2. Open Vswitch is also the de facto standard for SDN in the industry, and it is hoped that this will bring a variety of network SDN capabilities to the container, such as speed limit and QoS access control. It turns out that only when the container is created and deleted is called to Netplugin, all traffic in the running container only passes through open vSwitch, does not depend on netplugin, it can be accessed even if the container is hung, this mechanism is good for the reliability of the network. OvS has proven to be very stable in the previous 1.5 OpenStack practice, with OvS Bridge using a bandwidth of 1G uplink, with less than 5% loss compared to physical machines.

Netplugin The original scheme is a flow table, each new container will add a flow, and all nodes are added, the size of the container is not conceivable. We remove this function to reduce complexity and improve stability. In addition, the OvS rate-limit function is introduced, and the container flow control is also done, which can adjust the available bandwidth of each container in real-time according to the situation. The Netplugin managed IP address pool in the project is three, which supports the operation of the Explorer container well.

In order to prevent the growth of the same two-layer broadcast domain container, which causes the router ARP table to grow too fast, in the large-scale deployment, the ARP proxy function needs to be added in Netplugin.

Netplugin many good features such as Vxlan, multi-tenancy, access control we are not used. Although the community is growing, the code is not fully mature. Also encountered a number of bugs, such as the container unexpectedly exit the IP address can not release the problem, which needs our own to solve. Our practice is based on a certain version, thoroughly understand code, only with the basic functions, after full testing, edge measurement edge change, gradually expand the line scale.

About Container Storage

Container external volumes use convoy to support container persisted data in the form of plug-ins. Both the container itself and the external volume Use device mapper as the underlying. There is no reason to choose distributed storage, mainly to simplify the implementation and to be more stable. By limiting the blkiodevicereadbps, blkiodevicewritebps, Blkiodevicereadiops, and blkiodevicewriteiops of each container, the disk IO is stabilized to achieve the equivalent of 95% physical machine performance.

For device Mapper, because it is recommended by Red Hat, and the OS is a CentOS7.2, it is used. The testing process found that the device mapper robustness is not very good, only under low concurrency, there will also be a case of container deletion failure, the container and start-stop occasionally found no device. This kind of device uses the mapping relation, the function is rich, the implementation is too complex, each time changes to the device need to update metadata, the chance of concurrent scene error is big. Let me choose again, I will consider overlay this simpler driver.

For convoy, a product from rancher, go language, still in the immature stage, version number 0.5, and does not fully implement the volume plugin interface. It has the most problems compared to other modules, such as volume creation failure, unable to delete, UNIX socket leak, duplicate name conflict, exception Auto exit, etc. belong to the use, but not perfect state, you have to have a certain development and debugging capabilities to solve the problems found. Several other storage plug-ins are similar, Flocker, Blockbridge, Horcrux, and so on, some even the first official release has not yet, convoy is relatively good point, a bit rotten persimmon heap pick feeling.

For the volume plugin interface of Docker itself, we also encountered some pits. Here are some of them:

• Docker Volume Plugin does not support getting Volume usage state data
• Docker Volume Plugin exists file descriptor leaks bug--https://github.com/docker/docker/pull/20686
• Swarm regular list will accidentally trigger Docker volume cache bug-https://github.com/docker/docker/issues/21403

About container Monitoring

Container monitoring in this project can also have a lot of room to improve. The project is Cadvisor, container top, free, Iostat command hijacking, based on the existing Zabbix system for data collection and display. The conclusion is that Zabbix is completely unsuitable for container monitoring, data collection density, display quality, flexibility can not meet the demand.

Later in the test, try using Telegraf + InfluxDB + grafana. Just grafana simple configuration, can help us to clearly show the container and service process CPU, memory, network, disk and so on. The debug and development of SQL query statements on Grafana do take a lot of time, but this workload is one-off. Because it is go write, Telegraf CPU occupies a lower level (0.4-5%). Rich in functionality, while supporting the external process and container data collection, up to 55 kinds of data source plug-in, there is no need for cloth cadvisor, personal comparison recommended. Students who need to be alerted may consider changing the influxdb to Prometheus. It contains alertmanager to implement email, pagerduty and other message notifications. The data backend can choose the db that comes with, also can external influxdb, Graphite, Opentsdb and so on popular scheme.

Monitoring industry has a lot of open source solutions can be consulted, the following are the criteria to be measured: easy to expand, low cost, intrusion small, large concentration, easy to deploy, real-time, show clear and flexible. In this regard, we hope to have more exchanges with you.

Q&a

Q: There are now many ways to improve the Docker network with bridging bridges, can this be tested?

A: Bridging the bridge is a simple solution, but the IP address assignment can only be done in this machine ipam, not global management, there is a waste of address resources.

Q: What is the delivery of the test environment and pre-release environment in the field of the real-world development of the system?

A:mysql data backup and recovery capabilities.

Q: "The container unexpectedly exits the IP address problem, which needs to be solved by ourselves." "Can you provide a general solution?"

A: Plan to call a Releaseendpoint API via Libnetwork UNIX socket, which guarantees the integrity of the delete operation, including OvS Port, ETCD EP information, IP address.

Q:docker 1.12 What are the advantages of the built-in Docker swarm Mode,overlay network compared to other open source solutions?

A:overlay Network's advantage lies in the virtual network, multi-tenant scene support better, there is no 4,096 restrictions, but there is no support for Vxlan hardware without direct access, which for development, problem positioning, monitoring is a problem. Performance is not a bottleneck for containers.

Q: Did you do the Mesos 1.0 test? 1.0 no longer rely on Docker daemon?

Docker is still supported as a containerizer in A:mesos 1.0. Experimental environment verified that Mesos + Docker + netplugin is feasible, theoretically regardless of which version of the Mesos, as long as it still supports Docker, then you can network plug-in form to the ground network implementation.

Q: The container is generally ready to destroy, want to ask, you do the container monitoring, how to ensure data durability?

The data that is backed up by A:mysql is stored on the external volume of the container, on the host, and the container destroys the external volume and is not deleted, so the data remains intact.

Q: What are your container alarms based on, and cadvisor can't provide alarms?

A: Based on our existing monitoring system, that is Zabbix.

Does q:devicemapper use direct LVM? How can I expand my device space with full capacity?

A: Yes. Expansion can be achieved by adding physical disk devices, expanding the VG, and then resize the DM pool size.

Q: When you use hundreds of machines, is the image pre-downloaded well? Or temporary download, there is no mirror download to speed up this piece of the child?

A: The image is stored in the local IDC registry, all the machine access registry through the similar network, the download speed quickly. The good thing is to pull the global pull once more and run it.

Q: I am worried about the performance of the database in the container, you are so high performance, what is the main configuration?

A: Because we are afraid of competing for resources, we strictly limit the number of containers, CPU, memory, and disk of a single host. In the case of sufficient resources, MySQL running in the container and running outside is no essential difference, is the process.

Q: About monitoring this piece, mentioned above two, can you tell us about the division of labor between Zabbix and Cadvisor?

A:zabbix uses its own script to request container monitoring data to the Cadvisor restful interface.

What is the performance of the Q:mysql database after it is containerized? How fast is the failure recovery?

A: From the point of view of data backup recovery, the basic is equivalent to running on a physical machine.

The above content is organized according to the September 6, 2016 night group sharing content. Share Yeke, the only product cloud Platform Senior Development engineer. Ten years of experience in IT industry, the development of old drivers, growth in Ericsson, testing and development industry have fought. Technology implementation advocates "simple is the best", "not to solve the problem for the purpose of the introduction of technology is bullying." Now focus on the design and development of OpenStack, Docker and other cloud-related technologies, responsible for Docker's overall technical solution design, container network, monitoring, logging, operating system and so on, for the company's cloud computing technology infrastructure evolution to provide support. 。 Dockone organizes targeted technology sharing every week and welcomes students who are interested in adding: Liyingjiesz, participate in the group, you have to listen to the topic or want to share the topic can give us a message.