Create an efficient operational log collection and analysis platform

0x01 Background

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M02/77/C0/wKioL1ZtbXyh92g_AABPGn7B5Xo812.png "title=" Kibana.png "alt=" Wkiol1ztbxyh92g_aabpgn7b5xo812.png "/>

In the face of the increasing log information, the most traditional method of log collection is difficult to meet the basic needs of operation and maintenance personnel. So, why don't we use today's rich open source tools to create an efficient and useful operational log collection and analysis platform. The following is a brief introduction to the operation and maintenance log platform We are currently trying to do, and we hope to exchange experiences with you.

0x02 Platform Architecture

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/77/BF/wKioL1ZtaY6zpTEVAABJGFnJFHU134.png "title=" Elas-kafka.png "alt=" Wkiol1ztay6zptevaabjgfnjfhu134.png "/>

instead of using the Elk architecture for log collection, we used a variety of log collection tools, namely EKF (k/z),

Elasticsearch + kafka-zookeeper + Flume + Kibana/zabbix.

0x03 Brief Introduction    

     for flume Appropriate rewriting, the java Environment added to the Flume program to make flume When the service starts, you can have java environment, Eliminates the need to deploy the jdk

for kafka-zookeeper clusters, try to deploy in a Docker container. Docker is called for rapid deployment, so if you're deploying a larger number of kafka-zookeeper clusters, try using Docker deployment. First, we make a dockerfilethat will generate the Dockerfile image file and use it when it is started. We also encountered a lot of problems when deploying Kafka-zookeeper clusters using Docker. For example, the more prominent is the port problem, and finally we adopt a Docker container and host port mapping in a relatively simple way to solve.

650) this.width=650; "src=" Http://s2.51cto.com/wyfs02/M00/77/C1/wKiom1ZtbQqwcyy6AABF7jzU6Vg602.png "title=" Zook.png "alt=" Wkiom1ztbqqwcyy6aabf7jzu6vg602.png "/>

To write Kafka data to Elasticsearch, we use a middleware program that we write ourselves to implement. Originally wanted to use third-party plug-ins and other things, but found that suits their own is not, so, colleagues wrote a hand, the current verification, the effect and performance is good.

For Elasticsearch clusters, the current focus is mainly on how to optimize;

In addition, we will extract the data from the Elasticsearch and display it on the Zabbix. Currently, the parameters shown on Zabbix are mainly statistical and monitoring of PV values and HTTP status codes.

Zabbix

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M00/77/C0/wKioL1ZtbNPwRBmqAAB8BDFZwZc274.png "style=" float: none; "title=" Zabbbix1.png "alt=" Wkiol1ztbnpwrbmqaab8bdfzwzc274.png "/>

650) this.width=650; "src=" Http://s1.51cto.com/wyfs02/M01/77/C0/wKioL1ZtbNWxEL1eAACsKtq7sNE724.png "style=" float: none; "title=" Zabbix2.png "alt=" Wkiol1ztbnwxel1eaacsktq7sne724.png "/>

There is a lot more to be done for the operation and maintenance log collection and analysis platform. At present, just beginning, there are countless pits ahead, hoping to make a real log collection and analysis platform, simplifying the daily workload of operation and maintenance, to provide decision-making and processing basis for daily work.

(by Secscorpio)

Create an efficient operational log collection and analysis platform