Use Filebeat to transport logs for Docker containers

This is a creation in Article, where the information may have evolved or changed.

Today, let's talk about Docker container logs.

I. Old disease and capability evolution of container log output

By default, the Docker container stores log data that is printed to stdout, stderr, on the local disk, and the default location is/var/lib/docker/containers/{containerid}/{containerid}- Json.log. In the old version of Docker, this kind of logging is often criticized, such as: Unlimited log size, no Rotate (rotation), no log basic management capabilities, and poor performance. For these old diseases, Docker has been trying to refine and solve the evolution.

Memory seems to be in the Docker 1.8 version, Docker adds the rotate functionality to the Json-file (default) log driver, which we can configure with Max-size and Max-file two –log-opt. For example: We launch an nginx container, using the Json-file log engine, each log file limit is 1k, rotation of the number of logs is 5:

  $docker run-d--log-driver=json-file--log-opt max-size=1k--log-opt max-file=5--name webserver-p 9988:80 ng inx50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72$ sudo ls-l/var/lib/docker/containers/ 50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72 total dosage 44-rw-r--r--1 root root 226 March 14:39 50f100e7ea 4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72-json.log-rw-r--r--1 root root 1129 March 24 14:39 50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72-json.log.1-rw-r--r--1 root root 1130 March 24 14:39 50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72-json.log.2-rw-r--r--1 root root 1129 March 24 14:39 50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72-json.log.3-rw-r--r--1 root root 1129 March 24 14:39 50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72-json.log.4 ...  

With rotate, we don't have to worry about a container log spike that will drag the other container of the host to death. However, the management of logs has only evolved so far, and many needs have to be addressed by third-party tools and solutions.

In addition, the current performance of the Docker container log is still poor, if sensitive, can be resolved with the volume mechanism, that is, to close the application of the standard output, error (–log-driver=none), directly write the log to a mounted volume in a file. Below is a simple comparison of bare metal bare-metal native write log files, volume-way write log files, and docker default write JSON files:

We use DD this gadget to go1.6.linux-amd64.tar.gz this 85MB file as input, the result is as follows: (Environment Ubuntu 12.04 Docker 1.9.1)

1, Bare metaldd if=~/.bin/go1.6.linux-amd64.tar.gz Of=./go.bin recorded the 165623+1 read record of the 165623+1 write 84799480 bytes (MB) has been copied, 0.426716 seconds, 199 mb/sec 2, by hanging in local volume$ Docker run--rm-it-v/HOME1/TONYBAI/TESTDD/VOLUME:/TESTDD ubuntu:14.04 DD IF=/TESTDD /go1.6.linux-amd64.tar.gz of=/testdd/go.bin165623+1 Records in165623+1 records out84799480 bytes (MB) copied, 0.3753 s , 226 mb/s3, Docker default$docker run-v/HOME1/TONYBAI/TESTDD/VOLUME:/TESTDD ubuntu:14.04 DD if=/testdd/ go1.6.linux-amd64.tar.gz 2>&1 1>/dev/null165623+1 Records in165623+1 records out84799480 bytes (MB) copied, 5.97732 s, 14.2 mb/s$ sudo ls-lh/var/lib/docker/containers/ D4b5e6aae3968f68e5081414ad95c6308fa91808b44b415a03040403af5a4713/d4b5e6aae39 68F68E5081414AD95C6308FA91808B44B415A03040403AF5A4713-JSON.LOG-RW-------1 root root 331M March 18:05/var/lib/docker/                                         containers/d4b5e6aae3968f68e5081414ad95c6308fa91808b44b415a03040403af5a4713/ D4b5e6aae3968f68e5081414ad95c6308fa91808b44b415a03040403af5a4713-json.log 

As you can see, by default, the speed at which Docker writes to JSON is one-tenth lower than the mount volume method. The main reason is that the standard output of the Docker container, standard error will be taken over by Docker Daemon, and written by Daemon to the JSON log file, so daemon becomes the bottleneck of log writing.

II. Centralized management of container logs

Log management needs have been long-standing, whether traditional legacy systems, or Internet applications or services, the role of logs in the operation and operations can not be underestimated. In particular, the centralized log management practice, which is widely used now, puts forward new requirements for the log management of Docker. As mentioned above, with the evolution of Docker, the logging of Docker has improved, with the addition of multiple log driver support (such as syslog, FLUENTD, etc.), which provides a variety of centralized management solutions for container logs.

At present, many domestic enterprises adopt Elk scheme (Of course Elk solution is not only limited to Docker), that is Elasticsearch + Logstash + Kibana,logstash is responsible for collecting, filtering, analyzing and processing logs from each node, Elasticsearch is responsible for storing, indexing, and locating logs, and Kibana is responsible for displaying the results of log processing in a graphical interface. But Docker container how to do local log management, how to send the latest local log to Logstash there is no standard solution, you can use the FLUENTD agent can also be used logspout. The Elk scenario also has its own tool for client node log delivery, formerly known as Logstash-forwarder:

node1 (logstash-forwarder) ------>node2 (logstash-forwarder) ------>   logstash server --> ElasticSearchnode3 (logstash-forwarder) ------>

Now elastic.co use Beats series to replace Logstash-forwarder, for the log conveying this piece, the corresponding beats product is filebeat, using filebeat, the previous centralized log scheme structure becomes:

node1 (filebeat) ------>node2 (filebeat) ------>   [logstash server] --> ElasticSearchnode3 (filebeat) ------>

We see that Logstash server is an optional intermediate link, and after using Filebeat, you can send the latest log on client node directly to Elasticsearch without having to go through Logstash. Of course, if you have the source log filtering, cleaning, analysis, and other needs, Logstash is still your right-hand assistant. Here we do not logstash, but directly to the log sent to Elasticsearch for storage and indexing.

Iii. steps to Output container logs using Filebeat

The test environment is as follows: (Ubuntu 14.04 + docker 1.9.1)

node1 (10.10.126.101 nginx container +  filebeat) ------>   server 10.10.105.71 (ElasticSearch + kibana)

All the programs here are installed and run as containers.

1. Installation of Elasticsearch and Kibana

Both Elasticsearch and Kibana have official Docker image.

Install Elasticsearch:

$ docker pull elasticsearchUsing default tag: latestlatest: Pulling from library/elasticsearch...

Perform env to view version:

$ docker exec  elasticsearch env... ...ELASTICSEARCH_MAJOR=2.2ELASTICSEARCH_VERSION=2.2.1ELASTICSEARCH_REPO_BASE=http://packages.elasticsearch.org/elasticsearch/2.x/debian... ...

Install Kibana:

$ docker pull kibanaUsing default tag: latestlatest: Pulling from library/kibana... ...

Let's take a look at the current images list:

REPOSITORY                             TAG                 IMAGE ID            CREATED             VIRTUAL SIZEelasticsearch                          latest              c6b6bed19c45        8 days ago          347.1 MBkibana                                 latest              d2c9c3cfc682        12 days ago         295.4 MB... ...

2. Start ES and Kibana, verify service start OK

Start es:

$ sudo mkdir -p /data/elasticsearch$  docker run -d --name elasticsearch -p 9200:9200 -v /data/elasticsearch:/usr/share/elasticsearch/data elasticsearch4288b4db18af8575961faf940a1dc634fe30857bb184fb45611136b7bd3ffb7d

To view service startup conditions:

$ docker psCONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                              NAMES4288b4db18af        elasticsearch       "/docker-entrypoint.s"   21 seconds ago      Up 10 seconds       0.0.0.0:9200->9200/tcp, 9300/tcp   elasticsearch

The startup log is as follows:

$ docker logs elasticsearch[2016-03-24 11:00:29,289][info][node] [Katherine Reynolds] version[2.2.1] , pid[1], build[d045fc2/2016-03-09t09:38:54z][2016-03-24 11:00:29,291][info][node] [Katherine Reynol DS] Initializing ... [2016-03-24 11:00:29,863] [INFO] [Plugins]                      [Katherine Reynolds] modules [Lang-expression, Lang-groovy], plugins [], sites [][2016-03-24 11:00:29,894][info][env ] [Katherine Reynolds] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/disk/by-uuid/f5 77c0bc-665b-431b-96e0-e3536dc11469)]], net usable_space [114.5GB], net total_space [130.4GB], spins? [possibly], types [ext4][2016-03-24 11:00:29,894][info][env] [Katherine Reynolds] heap size [990.7m b], compressed ordinary object pointers [true][2016-03-24 11:00:31,881][info][node] [Katherine Reyno LDS] initialized[2016-03-24 11:00:31,881][info][node] [KatherineReynolds] Starting ... [2016-03-24 11:00:31,993] [INFO] [Transport] [Katherine Reynolds] publish_address {172.17.0.1:9300}, bound_addresses {[::]:9300}[2016-03-24 11:00:32,004][info] [ Discovery] [Katherine Reynolds] elasticsearch/d7hv_rchqa275xc7if1kkg[2016-03-24 11:00:35,058][info][clust Er.service] [Katherine Reynolds] New_master {Katherine reynolds}{d7hv_rchqa275xc7if1kkg}{172.17.0.1}{172.17.0.1:                     9300}, Reason:zen-disco-join (Elected_as_master, [0] joins received) [2016-03-24 11:00:35,075][info][http ] [Katherine Reynolds] publish_address {172.17.0.1:9200}, bound_addresses {[::]:9200}[2016-03-24 11:00:35,076][info] [ Node] [Katherine Reynolds] started[2016-03-24 11:00:35,144][info][gateway] [kather INE Reynolds] recovered [0] indices into cluster_state

Start Kibana:

The boot Kibana container needs to provide an environment variable parameter, the service address and port of es:

$docker run -d --name kibana -e ELASTICSEARCH_URL="http://10.10.105.72:9200"  -p 5601:5601 kibana

To verify that Kibana starts OK, you can open it through the browser: http://10.10.105.72:5601, if the Web page is displayed properly, and the Http://10.10.105.72:5601/status page has "Status: Green "indicates that the Kibana is starting OK.

3. Installation and Configuration Filebeat

Before installing Filebeat, we start a test with webserver, deployed on 10.10.126.101, to generate log data:

$ docker run -d --log-driver=json-file --log-opt max-size=1k --log-opt max-file=5 --name webserver -p 9988:80 nginx50f100e7ea4d5b4931f144f9eac12b6a05e56579583d7a0322b250004b68ae72

Filebeat There is no official image version, the largest number of star on the Docker hub is the image from the Prima/filebeat library, and we're going to use this, and the pull process doesn't go into this:

$docker run --rm prima/filebeat env... ...FILEBEAT_VERSION=1.1.2... ...

You can see that the filebeat version used by Filebeat image in this library is up-to-date.

Let's look at run Now:

$ docker run --rm prima/filebeatLoading config file error: Failed to read /filebeat.yml: open /filebeat.yml: no such file or directory. Exiting.

It seems filebeat need to do some configuration, we have to check the official manual of Filebeat.

Filebeat requires a FILEBEAT.YML configuration file to configure the log source and the destination of the log transport, we refer to the manual to provide a suitable configuration for us:

  filebeat: # List of prospectors to fetch data. Prospectors: # Each-is a prospector. Below is the prospector specific configurations-# Paths that should is crawled and fetched.      Glob based paths.      # for each file found under this path, a harvester is started. Paths:-"/var/lib/docker/containers/*/*.log" #-c:\programdata\elasticsearch\logs\* # Type of the FI Les.      Based on the the-the-the-file is read-is decided.  # The different types cannot is mixed in one prospector # # Possible Options is: # * Log:reads every line of the log file (default) # * Stdin:reads the input_type:log# Configure what outputs     Sending the data collected by the beat.# multiple outputs could be used.output: # # # Elasticsearch as output Elasticsearch:    # Array of hosts to connect to. Hosts: ["10.10.105.72:9200"]  

We collect/var/lib/docker/containers/*/*.log, which is the log of all containers in the node where filebeat resides. The location of the output is our Elasticsearch service address, where we send the log directly to Es instead of through Logstash.

Before starting, we also need to submit a Filebeat index template to es to let es know which properties and fields are included in the log data for the filebeat output. Filebeat.template.json This template file is not written by us, Filebeat is officially available, we can find it on github.com.

Load this template to es:

$ curl -XPUT 'http://10.10.105.72:9200/_template/filebeat?pretty' -d@/home1/tonybai/filebeat.template.json{  "acknowledged" : true}

If you see that the return result of Curl is true, then the load OK is indicated.

Next, we launch the Filebeat container:

$ docker run -d --name filebeat -v /home1/tonybai/filebeat.yml:/filebeat.yml prima/filebeatf93497ea816e5c4015e69376f98e791ca02b91a20145ee1366e4c15f6a706c10

Let's see if we can get a log of the container in the Kibana. When using Kibana, you need to add a new index pattern. According to the requirements of manual, our index name or pattern should be filled in as "filebeat-" for Filebeat,but I add default index:filebeat-in Kibana has failed, the following button has been gray, and prompted: "Unable to fetch mapping." Do you have indices matching the pattern ".

In Filebeat's forum to find the answer to the question, someone prompted: see if there is a elasticsearch in the filebeat transmission of the log. So look at the Elasticsearch log and through the API provided by Elasticsearch to make a query, found that filebeat there is no log transfer over.

Back to think carefully, wow, incredibly did not give filebeat container hanging in the/var/lib/docker/containers directory, then filebeat have no access to the container log, naturally there will be no log transfer to ES, the following output also confirms this:

$ docker exec filebeat ls /var/lib/docker/containersls: cannot access /var/lib/docker/containers: No such file or directory

Then modify the Filebeat startup parameters:

$docker run -d  --name filebeat -v /home1/tonybai/filebeat.yml:/filebeat.yml -v /var/lib/docker/containers:/var/lib/docker/containers prima/filebeat

After a while, we can successfully create the filebeat-* index pattern on Kibana and see the logs sent by Filebeat.

, Bigwhite. All rights reserved.