Using ETCD and Haproxy to do Docker service discovery

This is a creation in Article, where the information may have evolved or changed.

Using ETCD and Haproxy to do Docker service discovery

tags (space delimited): Etcd Haproxy Docker Service Discovery Architecture Discovery Docker-gen Golang Service

The author is Jwilder, the original text of this article is Docker Service Discovery Using Etcd and Haproxy

In the previous article, we showed a way to create an automated Nginx reverse proxy for the Docker container on the same host. That setting works well for front-end Web apps, but it's not a good idea for back-end services because they typically span multiple hosts.

This article describes a solution that provides service discovery for a Docker container for back-end services.

The architecture we will build is modeled after SmartStack, but using ETCD instead of Zookeeper, and two Docker containers running Docker-gen and haproxy instead of nerve and synapse.

How it works.

Similar to SmartStack, our Component Services as a registration (ETCD), a registered partner process (docker-register), the discovery partner process (docker-discover), Some back-end services (WHOAMI) and the last consumer (Ubuntu/curl).

The registration and Discovery component works as a device and application container, so the registration or discovery code on the backend or consumer container is not embedded. They only listen to ports or connect to other local ports.

Service Registration-ETCD

Before anything is registered, we need some place to track registration entries (for example, the IP and port of the service). We use ETCD because of the simple program model that is registered by the service and the TTLs and directories that support the keys.

Typically, you'll run 3 to 5 ETCD nodes, but we'll just use one to keep things simple.

There is no reason why we cannot use Consul or any other storage option to support TTL expiration.

Start Etcd:

docker run -d --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd

Service Registration-Docker-register

The Registration service container is handled by the Jwilder/docker-register container. This container registers other containers running on the same host to ETCD. The container we want to register must expose a port. Containers running the same image on different hosts are grouped in Etcd and will form a load-balanced cluster. How containers are grouped is a bit messy, for this walkthrough I have selected the container image name. In a real-world deployment, you might want to group by environment variables, service versions, or other metadata.

(The current implementation only supports one port per container and assumes that it is currently TCP, there is no reason why it cannot support multiple ports and types, and different grouping properties)

Docker-register uses Docker-gen along with a Python script as a template. When running, dynamically generate a script that will /backends register each container's IP and port in the directory.

Docker-gen is concerned about monitoring Docker events and invoking the Generate script at an interval call to ensure that TTLs is always on the nearest date, and if Docker-register is stopped, registration expires.

In order to start docker-register, we need to pass the host's external IP, and the other hosts can access its container and the address of your ETCD host. To call its Api,docker-gen requires access to the Docker daemon, so we also bind the UNIX socket attached to the Docker into the container.

HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}')ETCD_HOST=w.x.y.z:4001docker run --name docker-register -d -e HOST_IP=$HOST_IP -e ETCD_HOST=$ETCD_HOST -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register

Service Discovery-Docker-discover

Service discovery is handled by jwilder/docker-discover containers. Docker-discover periodic polling Etcd and generates a Haproxy profile by listening on each service type.

For example, the port on which the container runs Jwilder/whoami is registered /backends/whoami/<id> and exposed to the host is 8000.

Other containers need to call the Jwilder/whoami service and can send requests to Docker Bridge ip:8000 or host ip:8000.

If any of the backend services are down, the Haproxy health check removes it from the pool and attempts to request it on a healthy host. This ensures that back-end services can be started and stopped as demand, as well as the inconsistency in processing registration information, while ensuring a minimized client impact.

Finally, stats can be viewed by accessing the port 1936来 on the Docker-discover container.

Run Docker-discover:

ETCD_HOST=w.x.y.z:4001docker run -d --net host --name docker-discover -e ETCD_HOST=$ETCD_HOST -p 127.0.0.1:1936:1936 -t jwilder/docker-discover

We are using it so that the --net host container uses the host network stack. When this container is bound to port 8000, it is actually bound on the host's network. This simplifies the proxy settings.

AWS Demo

We will run a full suite of services on 4 AWS hosts: One ETCD host, one client host and two back-end hosts. The backend service is a simple Golang HTTP service that returns a host name.

ETCD Host

First, we start ETCD registration:

$ hostname --all-ip-addresses | awk '{print $1}'10.170.71.226$ docker run -d --name etcd -p 4001:4001 -p 7001:7001 coreos/etcd

Our ETCD address is 10.170.71.226 . We will use it on other hosts. If we are running an online environment, we can assign an EIP and a DNS address to it to make it easier to configure.

Back-end Host

Next, we start this service and docker-register on each host. The service is configured to listen on port 8000 in the container and we let Docker publish it to a random port on a host.

Back-end Host 1:

$ docker run -d -p 8000:8000 --name whoami -t jwilder/whoami736ab83847bb12dddd8b09969433f3a02d64d5b0be48f7a5c59a594e3a6a3541$ docker run --name docker-register -d -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -e ETCD_HOST=10.170.71.226:4001 -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register77a49f732797333ca0c7695c6b590a64a7d75c14b5ffa0f89f8e0e21ae47ae3e$ docker psCONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                     NAMES736ab83847bb        jwilder/whoami:latest            /app/http              48 seconds ago      Up 47 seconds       0.0.0.0:49153->8000/tcp   whoami77a49f732797        jwilder/docker-register:latest   "/bin/sh -c 'docker-   28 minutes ago      Up 28 minutes                                 docker-register

Back-end Host 2:

$ docker run -d -p 8000:8000 --name whoami -t jwilder/whoami4eb0498e52076275ee0702d80c0d8297813e89d492cdecbd6df9b263a3df1c28$ docker run --name docker-register -d -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -e ETCD_HOST=10.170.71.226:4001 -v /var/run/docker.sock:/var/run/docker.sock -t jwilder/docker-register832e77c83591cb33bba53859153eb91d897f5a278a74d4ec1f66bc9b97deb221$ docker psCONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                     NAMES4eb0498e5207        jwilder/whoami:latest            /app/http              59 seconds ago      Up 58 seconds       0.0.0.0:49154->8000/tcp   whoami832e77c83591        jwilder/docker-register:latest   "/bin/sh -c 'docker-   34 minutes ago      Up 34 minutes                                 docker-register

Client Host

On the client host, we need to start Docker-discover and a client service. For this client container, I use Ubuntu trusty and will make some curl requests.

First, start Docker-discover:

$ docker run -d --net host --name docker-discover -e ETCD_HOST=10.170.71.226:4001 -p 127.0.0.1:1936:1936 -t jwilder/docker-discover

Then, start a simple client container and pass it on to it HOST_IP . We are using the eth0 address, but we can also use the Docker0 IP. We are passing it an environment variable because it is configured to change between two deployments.

$ docker run -e HOST_IP=$(hostname --all-ip-addresses | awk '{print $1}') -i -t ubuntu:14.04 /bin/bash$ root@2af5f52de069:/# apt-get update && apt-get -y install curl

At this point, construct some requests to the WhoAmI service port 8000来 to see their load.

$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb

We can launch some instances on the backend:

$ docker run -d -p :8000 --name whoami-2 -t jwilder/whoami$ docker run -d -p :8000 --name whoami-3 -t jwilder/whoami$ docker psCONTAINER ID        IMAGE                            COMMAND                CREATED             STATUS              PORTS                     NAMES5d5c12c96192        jwilder/whoami:latest            /app/http              3 seconds ago       Up 1 seconds        0.0.0.0:49156->8000/tcp   whoami-2bb2a408b8ec5        jwilder/whoami:latest            /app/http              21 seconds ago      Up 20 seconds       0.0.0.0:49155->8000/tcp   whoami-34eb0498e5207        jwilder/whoami:latest            /app/http              2 minutes ago       Up 2 minutes        0.0.0.0:49154->8000/tcp   whoami832e77c83591        jwilder/docker-register:latest   "/bin/sh -c 'docker-   36 minutes ago      Up 36 minutes                                 docker-register

Then construct some requests on the client host again:

$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm bb2a408b8ec5$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 5d5c12c96192$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb

Finally, we close some containers and the routes are updated. This kills anything at the back end 2.

$ docker kill 5d5c12c96192 bb2a408b8ec5 4eb0498e5207$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 67c3cccbb8ba$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 736ab83847bb$ root@2af5f52de069:/# curl $HOST_IP:8000I'm 67c3cccbb8ba

If you want to see how haproxy is loading or monitoring errors, we can access the 1936 port of the client host in a Web browser.

Summarize

While there are different ways to implement service discovery, SmartStack's partner registration behavior and agents keep the application code simple and very easy to fuse into a distributed environment, really suitable for Docker containers.

Similarly, Docker events and container APIs mitigate the difficulty of service registration and the use of Registry services discovery (such as ETCD).

The code for

Docker-register and Docker-discover is on GitHub. Although both are useful, there are many places that need to be promoted. Please feel free to submit or make suggestions for improvement.