This is a creation in Article, where the information may have evolved or changed.
Using ETCD and Haproxy to do Docker service discovery
tags (space delimited): Etcd Haproxy Docker Service Discovery Architecture Discovery Docker-gen Golang Service
The author is Jwilder, the original text of this article is Docker Service Discovery Using Etcd and Haproxy
In the previous article, we showed a way to create an automated Nginx reverse proxy for the Docker container on the same host. That setting works well for front-end Web apps, but it's not a good idea for back-end services because they typically span multiple hosts.
This article describes a solution that provides service discovery for a Docker container for back-end services.
The architecture we will build is modeled after SmartStack, but using ETCD instead of Zookeeper, and two Docker containers running Docker-gen and haproxy instead of nerve and synapse.
How it works.
Similar to SmartStack, our Component Services as a registration (ETCD), a registered partner process (docker-register), the discovery partner process (docker-discover), Some back-end services (WHOAMI) and the last consumer (Ubuntu/curl).
The registration and Discovery component works as a device and application container, so the registration or discovery code on the backend or consumer container is not embedded. They only listen to ports or connect to other local ports.
Service Registration-ETCD
Before anything is registered, we need some place to track registration entries (for example, the IP and port of the service). We use ETCD because of the simple program model that is registered by the service and the TTLs and directories that support the keys.
Typically, you'll run 3 to 5 ETCD nodes, but we'll just use one to keep things simple.
There is no reason why we cannot use Consul or any other storage option to support TTL expiration.
Start Etcd:
docker run -d --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd
Service Registration-Docker-register
The Registration service container is handled by the Jwilder/docker-register container. This container registers other containers running on the same host to ETCD. The container we want to register must expose a port. Containers running the same image on different hosts are grouped in Etcd and will form a load-balanced cluster. How containers are grouped is a bit messy, for this walkthrough I have selected the container image name. In a real-world deployment, you might want to group by environment variables, service versions, or other metadata.
(The current implementation only supports one port per container and assumes that it is currently TCP, there is no reason why it cannot support multiple ports and types, and different grouping properties)
Docker-register uses Docker-gen along with a Python script as a template. When running, dynamically generate a script that will /backends
register each container's IP and port in the directory.
Docker-gen is concerned about monitoring Docker events and invoking the Generate script at an interval call to ensure that TTLs is always on the nearest date, and if Docker-register is stopped, registration expires.
In order to start docker-register, we need to pass the host's external IP, and the other hosts can access its container and the address of your ETCD host. To call its Api,docker-gen requires access to the Docker daemon, so we also bind the UNIX socket attached to the Docker into the container.
HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}')ETCD_HOST=w.x.y.z:4001docker run --name docker-register -d -e HOST_IP=$HOST_IP -e ETCD_HOST=$ETCD_HOST -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register
Service Discovery-Docker-discover
Service discovery is handled by jwilder/docker-discover containers. Docker-discover periodic polling Etcd and generates a Haproxy profile by listening on each service type.
For example, the port on which the container runs Jwilder/whoami is registered /backends/whoami/<id>
and exposed to the host is 8000.
Other containers need to call the Jwilder/whoami service and can send requests to Docker Bridge ip:8000 or host ip:8000.
If any of the backend services are down, the Haproxy health check removes it from the pool and attempts to request it on a healthy host. This ensures that back-end services can be started and stopped as demand, as well as the inconsistency in processing registration information, while ensuring a minimized client impact.
Finally, stats can be viewed by accessing the port 1936来 on the Docker-discover container.
Run Docker-discover:
ETCD_HOST=w.x.y.z:4001docker run -d --net host --name docker-discover -e ETCD_HOST=$ETCD_HOST -p 127.0.0.1:1936:1936 -t jwilder/docker-discover
We are using it so that the --net host
container uses the host network stack. When this container is bound to port 8000, it is actually bound on the host's network. This simplifies the proxy settings.
AWS Demo
We will run a full suite of services on 4 AWS hosts: One ETCD host, one client host and two back-end hosts. The backend service is a simple Golang HTTP service that returns a host name.
ETCD Host
First, we start ETCD registration:
$ hostname --all-ip-addresses | awk '{print $1}'10.170.71.226$ docker run -d --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd
Our ETCD address is 10.170.71.226
. We will use it on other hosts. If we are running an online environment, we can assign an EIP and a DNS address to it to make it easier to configure.
Back-end Host
Next, we start this service and docker-register on each host. The service is configured to listen on port 8000 in the container and we let Docker publish it to a random port on a host.
Back-end Host 1:
$ docker run -d -p 8000:8000 --name whoami -t jwilder/whoami736ab83847bb12dddd8b09969433f3a02d64d5b0be48f7a5c59a594e3a6a3541$ docker run --name docker-register -d -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -e ETCD_HOST=10.170.71.226:4001 -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register77a49f732797333ca0c7695c6b590a64a7d75c14b5ffa0f89f8e0e21ae47ae3e$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES736ab83847bb jwilder/whoami:latest /app/http 48 seconds ago Up 47 seconds 0.0.0.0:49153->8000/tcp whoami77a49f732797 jwilder/docker-register:latest "/bin/sh -c 'docker- 28 minutes ago Up 28 minutes docker-register
Back-end Host 2:
$ docker run -d -p 8000:8000 --name whoami -t jwilder/whoami4eb0498e52076275ee0702d80c0d8297813e89d492cdecbd6df9b263a3df1c28$ docker run --name docker-register -d -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -e ETCD_HOST=10.170.71.226:4001 -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register832e77c83591cb33bba53859153eb91d897f5a278a74d4ec1f66bc9b97deb221$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES4eb0498e5207 jwilder/whoami:latest /app/http 59 seconds ago Up 58 seconds 0.0.0.0:49154->8000/tcp whoami832e77c83591 jwilder/docker-register:latest "/bin/sh -c 'docker- 34 minutes ago Up 34 minutes docker-register
Client Host
On the client host, we need to start Docker-discover and a client service. For this client container, I use Ubuntu trusty and will make some curl
requests.
First, start Docker-discover:
$ docker run -d --net host --name docker-discover -e ETCD_HOST=10.170.71.226:4001 -p 127.0.0.1:1936:1936 -t jwilder/docker-discover
Then, start a simple client container and pass it on to it HOST_IP
. We are using the eth0 address, but we can also use the Docker0 IP. We are passing it an environment variable because it is configured to change between two deployments.
$ docker run -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -i -t ubuntu:14.04 /bin/bash$ root@2af5f52de069:/# apt-get update && apt-get -y install curl
At this point, construct some requests to the WhoAmI service port 8000来 to see their load.
$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb
We can launch some instances on the backend:
$ docker run -d -p :8000 --name whoami-2 -t jwilder/whoami$ docker run -d -p :8000 --name whoami-3 -t jwilder/whoami$ docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES5d5c12c96192 jwilder/whoami:latest /app/http 3 seconds ago Up 1 seconds 0.0.0.0:49156->8000/tcp whoami-2bb2a408b8ec5 jwilder/whoami:latest /app/http 21 seconds ago Up 20 seconds 0.0.0.0:49155->8000/tcp whoami-34eb0498e5207 jwilder/whoami:latest /app/http 2 minutes ago Up 2 minutes 0.0.0.0:49154->8000/tcp whoami832e77c83591 jwilder/docker-register:latest "/bin/sh -c 'docker- 36 minutes ago Up 36 minutes docker-register
Then construct some requests on the client host again:
$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm bb2a408b8ec5$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 5d5c12c96192$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb
Finally, we close some containers and the routes are updated. This kills anything at the back end 2.
$ docker kill 5d5c12c96192 bb2a408b8ec5 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 67c3cccbb8ba$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 67c3cccbb8ba
If you want to see how haproxy is loading or monitoring errors, we can access the 1936 port of the client host in a Web browser.
Summarize
While there are different ways to implement service discovery, SmartStack's partner registration behavior and agents keep the application code simple and very easy to fuse into a distributed environment, really suitable for Docker containers.
Similarly, Docker events and container APIs mitigate the difficulty of service registration and the use of Registry services discovery (such as ETCD).
The code for
Docker-register and Docker-discover is on GitHub. Although both are useful, there are many places that need to be promoted. Please feel free to submit or make suggestions for improvement.