A detailed approach to the design and implementation of Docker flat network

Research background

as we all know, Docker Container Cross-host visits have been a problem,Docker officials in order to avoid the network caused a lot of trouble, it will cross the host network opened a relatively large incision, and by the user to achieve. At present , there are many kinds of network implementations of Docker spanning host, including port mapping,OvS, Fannel and so on.

But none of these solutions can meet our needs: Intranet within the Port mapping service IP is mapped to an extranet IP, which can be confusing for development because they often do not require an intranet IP when interacting across networks , while OvS and Fannel is to wrap a layer of custom protocol on the basic network protocol, so that when the network traffic is large, it increases the network load unnecessarily. Finally, we took the self-developed flat network plug-in, which means that all the containers in the sophomore layer of interoperability.

Docker native four types of network mode

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > docker There are many kinds of network patterns, and then it's simple to Bridge , Host container, none

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > a. Bridge mode This mode is Docker Docker daemon  A default bridge is established on the host computer Docker0,  > believe it. Docker0 very familiar, but it is not useful in cross-container communication because the default docker0 docker0 Docker A disadvantage of native default network. Of course, he realized in the yourselves host network isolation, have their own namespace,  Nic and IP

B. Host mode

This mode is actually to share the network space with the current host, and Docker itself is not network isolation, said the popular point, that is, the container is in fact, and the host has the same IP, and how to specifically distinguish between the various containers? That is, by port mapping, Specify the - p parameter when starting the Docker container to set the port mapping. Although this approach can be used to some extent to achieve the purpose of cross-host container access, but it loses the meaning of Docker network isolation, and port mapping also brings some trouble to the microservices migration, not like the non-virtual environment as smooth migration, Instead, consider the issue of many port conversions.

C. Container mode

As the name implies, this mode shares the network namespace of another container, but it is limited to one host and still cannot implement the ability to communicate across hosts between containers.

D. mode None

The pattern is that the container has its own network namespace, its own web stack, its network card, no connection with the outside world, the container network is completely independent, in other words, the container does not require network functions, this mode for the container contains write data to disk volume of some tasks. This mode still fails to implement the functionality of our cross-host container network communication.

self-research Docker Overlay Network mode

currently Overlay Network mode is mainly implemented by tunneling and routing, one is to packet the basic network protocol, and the other is to configure more complex routing configuration to realize the network communication among the containers across the host. In fact, the above two more or less will give us the implementation of the network to bring complexity and loss of performance, because when we have a large business cluster, these complexity and performance loss can not be ignored.

The plugin principle is as follows :

1. creating a Docker custom network

Docker Network Create

--opt=com.docker.network.bridge.enable_icc=true

--opt=com.docker.network.bridge.enable_ip_masquerade=false

--opt=com.docker.network.bridge.host_binding_ipv4=0.0.0.0

--opt=com.docker.network.bridge.name=br0

--opt=com.docker.network.driver.mtu=1500

--ipam-driver=talkingdata

--subnet= The subnet range of the container IP, example:172.18.0.0/17

--gateway=br0 Bridge uses the IP, that is , the host address, example:172.18.0.5

--aux-address=defaultgatewayipv4= The gateway address used by the container mynet

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > We first need to create a br0 Custom bridge, this bridge is not manually created by the system commands the original linux bridge, but through docker Create Network command to build a custom bridge, thus avoiding a very important problem is that we can set the defaultgatewayipv4 self-built bridges cannot solve the problem .  with Docker subnet ip< Span style= "font-family: the song Body;" > range, by default we can give the entire network segment to this subnet, which can be controlled with ipam driver (address management plug-in). There is also a parameter gateway is used to set br0 Custom bridge address, It's actually the address of your host.

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > 2.ipam

This driver is dedicated to managing docker < Span style= "font-family: the song Body;" > container Ip , Docker  ip ip ip address etcd There is an operation to add and delete the changes. This plug-in runs with a Unix socket,  Docker/run/plugins . Sock files, Docker daemon sock  file to communicate to invoke the several interfaces we have implemented before Ip ip ip conflict.

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > 3.

through docker br0 /etc/sysconfig/network-scripts/ ifcfg-br0, ifcfg- Default network interface name) bridge the default network interface to br0 After restarting the network, the bridging network takes effect. docker br0 br0 docker  The container and host physical network interfaces are servers, through veth Pair This network device is like a cable plugged into the switch. At this point, all the container networks have been able to communicate on the same network, each docker ip< Span style= "font-family: the song Body;" , you can achieve cross-host access.

4.etcd

we can set1,3,5,7node is aEtcdcluster to centralize managementDockerof the clusterIp, and we also managed to manage the host's address in a unified way to avoidIpusage conflicts cause online resources to be unavailable. We will use our own tools to developIpinitialized, which means that it will pass in aIprange, and then the tool will allIpDeposit inEtcd, each networkIdis aEtcdDirectory , the directory will be divided into assigned and unassignedIpThe address pool. Etcditself will provideGolanguage ofApito accessEtcdcurrentlyEtcdis still quite stable, no problems have arisen.

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > epilogue

< Span style= "FONT-FAMILY:CALIBRI;FONT-SIZE:16PX;" > our docker cluster is using swarm relative k8s docker 1.12 after release , swarm will be integrated into docker Inside, and it adds a lot of functionality, I think the problem will be solved by then. Manage Swarm shipyard cpu BUG in the number of cores per container. yarn cluster, network performance is no problem.

Article Source: InfoQ

A detailed approach to the design and implementation of Docker flat network