Using Openvswitch to build a multi-host Docker Network "editor's note" When you successfully run a Docker container on a single host, and confidently plan to extend it to more than one host, you find that the previous attempt is equivalent to writing a Hello World starter program, A multi-host network setting is the next hurdle. When you try a variety of solutions, you may want to look at this article, perhaps will be enlightened, found that the original is not complex. Well, yes, this article uses the Openvswitch.
Running Docker is nothing new, there are many introductory tutorials on the Web to help you run containers on a single host. This host can be a Linux server or a Mac (with projects like Boot2docker).
Running on more than one host is another thing ...
Optional options:
- Run Docker on each host, exposing ports on a public or intranet network card for communication between containers. This can be cumbersome and can lead to security issues.
- Run a middle-tier scenario like weave to completely abstract the network. The project is promising, but it is too young to be integrated with compose (previous fig) or maestro-ng-like orchestration tools.
- Run a docker multi-host one-stop solution like Deis or Flynn. This may not be in your mind.
- Create a shared bridge in a mesh network between hosts to let the Docker service run the container. It sounds a little complicated, but ... In this article we will see that this can be done very easily!
Overview Basically, we will perform the following steps:
- Install Docker on each server;
- Install Openvswitch on each server;
- Custom network settings to automatically create bridges/tunnels between hosts (in each server
/etc/network/interfaces );
- Customize each Docker service configuration to handle only
docker0 a small portion of the IP range to prevent overlapping of the IP addresses of the new containers.
That's it. After restarting the service or restarting the server, you will get a fully meshed network with connected redundancy (link redundancy), which can run containers on a dedicated IP range (not overlapping) and can be interconnected without exposing all ports on a public or intranet network card. It's great, right?
Technology simply lists the technologies we use:
- Docker: Well ... This is an article about Docker and the Web, so ...
- Openvswitch: Very good virtual network switch project, scalability is excellent, according to this guide, you can run "any" scale of the network.
We will assume that the server is running Ubuntu Server 14.04.02 LTS x64, and for other systems you may need to modify the configuration provided below.
Installing Docker doesn't have to be a lot of talk, follow the guidelines provided on the website. We'll dive into its configuration later so that the different Docker services running on the server can collaborate with each other.
Openvswitch Unfortunately, the Openvswitch installation package is not available (or expired) in the default repository and we need to build it ourselves
.debFile (once) and distribute it to different hosts. In order to keep the production machine tidy, you can find another small host to install the development package, and build the installation package.
A detailed build manual is available on Openvswitch GitHub.
Execute the following command to build the installation package (please modify the new version as required):
#获取最新存档wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gztar xzvf openvswitch-2.3.1.tar.gzcd openvswitch-2.3.1# Installation Dependent sudo apt-get install -y build-essential fakeroot debhelper autoconf automake bzip2 libssl-dev openssl graphviz python-all procps python-qt4 python-zopeinterface python-twisted-conch libtool# build (Do not use parallel check) deb_build_options= ' Parallel=8 nocheck '  FAKEROOT DEBIAN/RULES&Nbsp;binary# get the latest Deb file and copy it to somewhere cd . Ls -al *deb
Now that you have a new
.debInstall the package, and then push it and install it on all hosts.
# Copy package to each host and SSH login scp-r *deb [email protected]_host:~/.ssh [Email protected]_host# Install some dependencies (required later) and install package sudo apt-get install-y b Ridge-utilssudo dpkg-i Openvswitch-common_2.3.1-1_amd64.deb Openvswitch-switch_2.3.1-1_amd64.deb
Configure the network you can use the different command-line tools provided by Openvswitch to build a mesh network (e.g.
ovs-vsctl), but Ubuntu offers an assistive tool that allows you to
/etc/network/interfacesThe file defines the network.
Assume three hosts: 1.1.1.1, 2.2.2.2, and 3.3.3.3, can ping each other through the above IP, they are not important in the public network or intranet. Host1 's
/etc/network/interfacesProbably as follows.
... # eth0, eth1 and lo configuration ... # auto: To effectively start it when the host starts # br0=br0: Prevent being found at ' ifquery --list ' auto br0= br0allow-ovs br0iface br0 inet manualovs_type ovsbridgeovs_ports gre1 gre2ovs_extra set bridge ${iface} stp_enable=truemtu 1462# No auto, this is an extra configuration for OvS # Two hosts must match the GRE name Allow-br0 gre1iface gre1 inet manualovs_type ovsportovs_bridge br0ovs_extra set interface ${iface} type=gre options:remote_ip=2.2.2.2allow-br0 gre2iface gre2 inet manualovs_type ovsportovs_bridge br0ovs_extra set interface ${iface} type=gre options:remote_ip=3.3.3.3# auto: Create # at startup Define the DOCKER0 that Docker will use, and (when available) connect to the Br0 bridge created by Openvswitch # each host needs to use a different IP address (do not conflict!). ) auto docker0=docker0iface docker0 inet staticaddress 172.17.42.1network 172.17.0.0netmask 255.255.0.0bridge_ports br0mtu 1462
Make some adjustments to this configuration on other hosts:
remote_ipIP addresses to be paired with each other.
A few notes:
- Spanning Tree Protocol (Spanning tree Protocol): If you apply this configuration, you will create a network loop in 3 servers.
br0adding to the bridge stp_enable=true will ensure that some gre tunnels are cut off. Also ensure the redundancy of the mesh network, allowing the network to recover when one of the hosts is offline.
- MTU: This is a key setting! Without this, you might get some unexpected "surprises": The network looks fine (for example, it can ping), but it can't support big packets (such as iperf in BW tests, big data requests, or simple file copying). Note that GRE tunnels need to encapsulate multiple protocols:
- Ethernet: 14 Bytes--We're talking about the 2nd layer between the bridges;
- Ipv4:20 bytes--Communication between containers/hosts;
- Gre:4 bytes--because, well, this is a GRE tunnel;
- That is, the physical Nic MTU minus 38 bytes, the result is 1462 (based on the regular MTU NIC).
- Use "" in Auto definition
= : This is not required for servers with fixed IP, but some cloud service providers (don't say who it is ...). Digital Ocean (Translator: The soft-wide re-entry)) uses a dependent ifquery --list --allow auto init service ( /etc/init/cloud-init-container.conf ). The " = " number will contain the Openvswitch NIC and delay the entire boot process until the Init script fails and times out.
- DOCKER0 Bridge: Each server needs its own IP address (for example
172.17.42.1 , 172.17.42.2 ). Because the bridges are docker0 br0 above the bridge, they will (and should!) ) can be connected to each other. Imagine how messy it is to resolve IP collisions ... That's why we define it at startup without relying on the Docker service to create this bridge for us.
- GRE tunnel: You can start with gre0 (not gre1) and it works perfectly. But for some reason,
ifconfig you can see it when you enter gre0 it, but you can't see the other tunnels. This could be gre0 a side effect of being a virtual NIC. From the gre1 beginning, all the gre tunnels will be ifconfig "invisible" (so much so that only one can be seen). Don't worry, you can still use the ovs-vsctl command to display the tunnel/bridge.
- 3 + hosts: You can follow the same logic, and:
- Add additional tunnels (Iface Grex) to connect to the new host.
- Updated in the
br0 bridge definition ovs_ports to include interfaces all the tunnels defined in the file gre .
- Be smart ... Do not link each server to another host one by one ... STP Convergence (convergence) will take longer and cannot provide any useful value other than multiple additional link redundancy.
If you restart the server now, you will have a redundant mesh network, and you can run the following command to test:
- From the host1
ping 172.17.42.2 or other IP;
- Run on the host
iperf by ifconfig viewing the links in use;
- When you ping the third host, stop the "middle", and look at the network convergence (through STP) when the ping is interrupted for a few seconds.
Docker we now have a well-established network where each Docker service can hook up their containers to
docker0The bridge. Wouldn't it be nice to have Docker do this automatically? The answer is that Docker has the ability to allocate a minimal pool of IP addresses!
For this example, we assume that:
- Each host (
1.1.1.1 ,, 2.2.2.2 3.3.3.3 ) is attached to the docker0 Network Bridge created earlier, its respective IP address is 172.17.42.1 , 172.17.42.2 , 172.17.42.3 ;
- Assign
docker0 A/16 IP range to the NIC;
- A small chunk of IP range is assigned to each host
docker0 and fixed-cidr saved in their Docker service configuration as/18. Respectively is 172.17.64.0/18 , 172.17.128.0/18 , 172.17.192.0/18 .
If you have more than 3 hosts, you need to subdivide each scope or reconsider the entire network topology based on your organization's needs.
Configuration file for Host1 (
/etc/default/docker) Is this:
Bridge=docker0cidr=172.17.64.0/18wait_ip () {address=$ (IP add show $BRIDGE | grep ' inet ' | awk ' {print $} ') [-Z ' $address "] && Sleep $ | |:}wait_ip 5WAIT_IP 15docker_opts="-H unix:///var/run/docker.sock-h tcp://0.0.0.0:2375-- Fixed-cidr= $CIDR--bridge $BRIDGE--mtu 1462 "
You can modify it as needed
DOCKER_OPTSConfiguration, add mirroring, insecure registry, DNS, and more.
Description
- WAIT_IP: Because
docker0 the Network bridge was finally created, it may take some time to get the IP address. With wait_ip features, you can safely wait a few seconds before returning to the Docker init script. The configuration file is referenced by the real init script ( /etc/init/docker.conf ).
- MTU: For the same reason as before, it is only a precaution to ensure that each NIC is created with the correct MTU specified.
- -H tcp://... : If you don't want to
0.0.0.0 "expose" it (or bind it to one of the server's "real" NICs), you can also securely bind it to ... The IP address of the host docker0 (for example 172.17.42.2 )! This allows you to access any of the Docker services in a private mesh network from any host computer.
The epilogue is restarted (at least everything will be automatically launched when it starts).
You can try the following commands to see if everything is OK.
# access Host1ssh [email protected]# run a new container docker run-ti ubuntu bash# check IP (run inside container) IP Add | grep eth0## in other Windows # # access another host (HOST2 or 3) SSH [email protected]# run a new container docker run-ti ubuntu bash# ping other containers! Ping $IP
This is not an authoritative guide on how to set up Docker on multiple hosts, and you are welcome to criticize (Translator note: The same as the translation, please correct me). Many ideas are generated during the overall installation, and this article explains in as much detail as possible why this or that option was chosen.
If you include a tiered bridge, VLAN, and so on, things will be more complicated, but that's beyond the scope of this article. ;)
Clearly, a more complete network is needed, and it looks like this is already under development.
Reference content
- https://goldmann.pl/blog/2014/... osts/
- http://networkstatic.net/open-... tion/
- Http://networkstatic.net/confi ... itch/
- Http://fbevmware.blogspot.com. ... . Html
- Http://openvswitch.org/support .... Html
- Https://access.redhat.com/docu .... Html
- Https://communities.vmware.com ... ation
- Http://www.microhowto.info/tro .... Html
- http://blog.scottlowe.org/2013 ... itch/
- http://blog.scottlowe.org/2013 ... itch/
- Https://github.com/openvswitch ... an.md
- http://baturin.org/tools/encapcalc/
If you have finished reading this article, congratulations!
Docker Multi-host scenario
