Build a docker environment for the Distributed log platform from the beginning and build a docker

Build a docker environment for the Distributed log platform from the beginning and build a docker

 

In the previous article (spring mvc + ELK build a log platform from the beginning), we will share with you how to build a distributed log Platform Based on spring mvc + redis + logback + logstash + elasticsearch + kibana, it is operated on the windows platform. This article mainly involves all these software environments in linux + docker.

Our goal is to install software environments such as resis and elk in docker. Because I am a windows platform, I searched for boot2docker, But I was prompted to use dockermachine for installation, dockermachine works with Oracle VM VirtualBox to create a linux virtual machine automatically when a docker is created. After comparison, I finally decided to install docker in the linux System in the traditional way. I chose VMware Workstation Pro for virtual machines and the latest ubantu for the operating system. After installation, I still need to perform many operations, such as disk ing and window size resolution, paste and copy support and so on to facilitate the use of virtual machine settings.

With a virtual machine like a machine, you can start working on docker. It is relatively simple to install the docker environment. We only need to follow the steps in this link (Installation on Ubuntu, although you are not familiar with some operation commands, you do not need to be overly nervous. We only want to install them successfully. If you are interested, you can study the purpose of each command in detail. In this article, you need to execute the corresponding commands based on your installed ubuntu version. The new version is simpler, and the test method is attached after the installation is successful. Run docer version. If you see the docer version as shown in, the installation is successful.

Found in the plugin, which contains detailed operation steps. The following describes some problems encountered during the installation process.

If you find that some commands are unclear during execution, the first method is to go to the official website, and the second method is to use-h or -- help to view them in the command.

• Redis

docker run -v /myredis/conf/redis.conf:/usr/local/etc/redis/redis.conf --name myredis redis redis-server /usr/local/etc/redis/redis.conf

I wrote the following version according to the above command:

docker run -dit --name redis  --hostname  redis-host  -v /data:/data -p 6379:6379 redis redis-server /data/redis.conf

The preceding command causes docker to execute an exception. The problem lies in the configuration file setting. You modified the path for loading the configuration file and changed it to the following, I do not know whether the path of the configuration file does not support arbitrary modification. The following command does not specify the redis-server and configuration file. Initially, it seems that the configuration file will be loaded by default and will be verified in the future.

docker run -dit --name redis  --hostname  redis-host  -v /data:/data -v /data/redis.conf:/usr/local/etc/redis/redis.conf -p 6379:6379 redis

• • -P indicates port ing. The port inside docker is mapped to the host port. For example, when multiple resis instances are installed, this port ing must be set, otherwise, the port on the host is occupied.
  • -- Name, which is used to conveniently operate docker Based on the docker name. Otherwise, the operation can only be performed based on the generated ID, which is not intuitive. For example, we can start redis using docker start.

• • -- Hostname: The name displayed after entering docker. For example, the name displayed after executing docker exec-it redis bash.

• • -V is the disk path ing, and the path relationship between docker and host
  • -D is run in the background mode. Other I and t can be used to query documents.
  • If the image version is not specified, it will automatically find the latest version for installation.
• Logstash

docker run -dit --name logstash  --hostname  logstash-host -v /data:/data logstash logstash -f /data/logstash.conf

The installation of logstash does not result in a trigger of redis. You can directly specify the path of the configuration file. The problem lies in the configuration of the configuration file. The following is a normal configuration.

input {    redis {        data_type => "list"        key => "logstash"        host => "192.168.21.128"        port => 6379        threads => 5        codec => "json"    }}filter {}output { elasticsearch {        hosts => ["192.168.21.128"]    }    stdout{}}

The configuration in the previous windows version is described, but the configuration in docker is different. You can directly retrieve the configuration file in windows, and logstash cannot output the logs in redis to elastcsearch, the output configuration is as follows:

output {       elasticsearch {        hosts => ["127.0.0.1:9200"]        index => "logstash-%{type}-%{+YYYY.MM.dd}"        document_type => "%{type}"        workers => 1        flush_size => 20        idle_flush_time => 1        template_overwrite => true    }    stdout{}}

The following two aspects are analyzed:
Input: Check whether the logback persists the log to redis. You can use the key * to check whether there is a logstash key specially configured for the log, then, use llen logstash to test whether the log increases with the execution of the program. The result is that the key of logstash can be found and the length increases with the execution of the program, the problem is that logstash does not take the redis logs and output them to elasticsearch. It deletes the elasticsearch configuration in the output and only keeps the stdout, the result shows that logstash can normally take logs away.

Output: The Troubleshooting of input indicates that the problem lies in output. Then, the least configuration method is adopted to configure only the most important attributes, and the hosts is configured in the document before running. As a result, the program runs normally.
We follow.

• Elasticsearch

docker run -dit --name elasticsearch  --hostname  elasticsearch-host  -v /data:/data -p 9200:9200 -p 9300:9300 elasticsearch:latest

The installation of elasticsearch is strange. After the above command is installed, access port http9200 on the client and prompt that access is denied. The port is displayed as tcp. After numerous installations, the installation is successful once accidentally, the port is mapped normally. The detailed problem is not found yet, so I will try again later. We can see the results after docker ps. Normally, the port is displayed as shown in the following figure. If a problem occurs, the port is not displayed as follows: 0.0.0.0: 9200-> 9200/tcp, 0.0.0.0: 9300-> 9300/tcp, but 9200/tcp, 9300/tcp.

• Kibana

docker run -dit --name kibana  --hostname  kibana-host  -v /data:/data -p 5601:5601 -e ELASTICSEARCH_URL=http://192.168.21.128:9200  kibana:latest

The Command Execution of kibana also encountered a problem specified by the configuration file. Later, the attribute was directly placed on the command, which is also a problem to be solved in the future.

After the above four items are successfully installed, our logs can be successfully collected to elasticsearch.


After the above docker is started, it does not have the self-starting function and can also be done. If the self-starting function is not provided, we cannot run one command to restart it. Here, a command can be used to start docker all at once, which is very practical.

docker start `docker -ps -a -q`