This is a creation in Article, where the information may have evolved or changed.
"Editor's note" This article mainly describes how to manage container logs in a container cluster, and what methods can be used to manage the logs, and what are the advantages and disadvantages of each of these methods. Then we discuss the current network of several schemes, each of which will bring the benefits and effects. Finally, the best practice of mango TV is introduced.
Let me first assume that tonight's audience is at least a small-scale rollout of Docker containerized technology online, at least familiar with how Docker works and the remote API. So I'm not going to introduce the basic operation and use of Docker, mainly to share the cluster container
Log management and network management。
In the early Docker implementations, the log was not fully functional, and standard output and errors in all containers were written to
/var/lib/docker/containers/{$cid}/{$cid}-log.jsonIn Since no logs are automatically divided and the container is bound, there will be an instantaneous disk full on the line. This file is also the data source for the Docker logs API, plus the Log-driver parameters introduced by Docker 1.6, so there are several ways we can manage the collection of online logs.
monitor files and transfer data through the pipeline . The biggest problem with this scenario is that the log files and containers are bound, so you need to have an agent role to do this, in disguise, increase the cost of development, but also consider the reliability of the pipeline. In addition to the CentOS 6 series and 7-series log address is not the same, if hard coding is poor scalability, if you read the system configuration, it is necessary to consider the cross-system path problem.
docker logs api remotely redirect logs by . The biggest problem with this method is that you can't avoid it or you have to have an agent to clean up the log so that the disk will still be full, of course, you can also cooperate with logrotate to do this, but increase the operation and maintenance costs. If the remote call this API, you need to consider the reliability of the connection, once the reconnection, it is necessary to do log backtracking, or you will lose a portion of the log.
The process in the container writes its own log . There are two options, as follows:
- Process directly write, control to the business side, the business is not transparent, controllable, after all, is the cluster environment. This also exposes the cluster structure to the upper layer.
- Mapping log device (/dev/log) into the container, the container process directly write the device, the isolation is weakened, a single point of problem tracking will be very troublesome, because this time stdout and stderr is no content, that is, the
docker log command without any output.
"Process directly write" This scenario we tried, but to let the business party to change the code, so the overall promotion is very difficult. In addition, it exposes the remote log server address, both on the network and on security issues. For example, once involved in the way of managing networks such as SDN, it is tantamount to destroying the isolation of the whole. "Mapping log devices into the container" This solution is to locate the problem container is more troublesome, but also involves communicating with the business parties.
using the new version of the Log-driver parameter , which includes support for syslog, looks pretty good, but in a clustered environment it is important to consider a single point of syslog. In general there will be more than one syslog or remote server (Logstash) that supports the Syslog protocol. If you use a remote syslog to accept logs, a large number of container log outputs are not averaged, resulting in performance hotspots and traffic hotspots. If you go to the single-machine syslog, and then summarized, and the above program "process direct output" Not much difference, the same is the problem of tracking more trouble. I think the current implementation of this is more convenient before using the syslog scheme.
The container output stream redirection is intercepted by the Attach method . This kind of solution needs agent support and has certain development requirements. Currently we are using this scheme, through a module to achieve the consistent hash, and then the log traffic to the remote collection server. This solution only needs to allow the business to output logs to stdout/stderr, without increasing development costs. At the same time Docker 1.6 can specify that the log driver is none, avoiding the generation of logs files. On the other hand, the container's own meta info can be attached to the log stream, so that the remote log retrieval classification and aggregation operations. But the biggest problem of this solution is the input of the development force, different dockeclient realize the quality is not the same, of course, the benefits are also obvious, flexible and controllable, log flow and distribution are injured in their own.
So the log aspect, from the current Docker implementation, if the development force to keep up, Agent + Attach program is the highest flexibility and controllability. At present log-driver for the scale of the cluster is still not very good, under the ideal state I would like to be able to specify multiple log-drivers, through the hash scheme to hit the far end. Of course, the choice of specific options depends on the respective company's own infrastructure and design objectives.
Say the log for the network, the current Docker network solution There are so many, of course, now everyone is waiting for 1.7, but I think for the production system, there is no too much care about the SDN solution Libnetwork, may be studied under its and how Docker through plugin Way of combining. Because the other case is now the Hook way to do it.
- Default Nat/br/host,nat has a performance loss, BR has a network flash, HOST flow control is not good to do, port conflict by business assurance can not be transparent.
Network layer Scenarios
A. Tunnel programme
I. OVS, mainly in terms of performance losses, based on VxLAN and GRE protocols, similar schemes have Kubernetes/socketplane implementations.
II. Weave, UDP broadcast, this machine set up a new BR, through PCAP interoperability.
Iii. Flannel, UDP broadcast, VxLan.
Tunnel scheme is very flexible, but because too flexible, out of network problems (A-B link jitter) tracking is more troublesome, large-scale cluster situation this is a point to consider, after all, even if the intranet is not necessarily calm.
B. Routing Scenarios
I. pipework, the expansion of the Docker BR itself, of course, also supports OVS Macvlan and other schemes. Now libnetwork out in disguise is the abolition of the project, in the long run no one, so it is not a good choice.
II. Calico, a BGP protocol-based routing scheme that supports granular ACL control, is suitable for scenarios where isolation requirements are more stringent, and has a higher affinity for hybrid cloud because it does not involve two-tier support.
III. Macvlan, from the logical and Kernel layer of isolation and performance of the best solution, based on two layer isolation, so requires two layer router support, most of the cloud service providers are not supported, so hybrid cloud is more difficult to implement.
Routing scheme is not so flexible, most cases need an agent on the container host to operate, generally from 3 or 2 layers to achieve isolation and cross-host container interoperability, out of the problem is also easy to troubleshoot. However, the routing scheme is heavier than the tunneling scheme for its own physical network dependency. The other hooks are not so beautiful after all, so we have to see how libnetwork is combined with Docker.
Currently
we chose the two-layer isolation of the Macvlan scheme implementation., said that the light point is mainly for the container, can be completely when the container for a virtual machine. The focus is on the fact that it relies most heavily on the physical network infrastructure.
Q&a
question: How did Macvlan do it?
Answer:Linux kernel support, directly with the agent on the host build device is plugged into the container namespace.
question: Why not just hit the log?
Answer:The container must consider the Docker itself storage performance issues, if the container is outside to consider volume management asked questions, if it is far-end, you have to consider and business integration issues.
question: Is the network layer a more mature choice?
Answer:Macvlan into the core, Calico did 10 years, the 2 will be more mature, weave relatively simple and convenient.
problem: Log loss issues
Answer:As far as possible to prevent the business layer to control the log output, even if the socket file or device can be deleted resulting in log loss, so as far as possible from the platform level control.
problem: Binary services running in Docker
Answer:At the moment we're doing something similar, running Redis. However, because the container itself does not have an init process, kernel parameters are default, and some binary applications are more sensitive in this regard. We have not found a more graceful way to solve it at present. There are many similar issue in the official Docker warehouses.
===========================
The above content is organized according to the May 5, 2015 Night Group sharing content.
Pengjove, head of the core technical team of Mango TV platform, is responsible for the development of Docker and Redis cluster related technical facilities. Previously served as the main course of the Watercress app engine and Jinshan fast disk. Have a lot of experience in system engineering. Next, Dockone organizes the targeted technology sharing every week, welcoming the interested students plus me (LIYINGJIESX) to participate.
