Elasticsearch + logstash + kibana build real-time log collection system "original"

Benefits of the unified collection of real-time logs:

1. Quickly locate the problem machine in the cluster

2, no need to download the entire log file (often relatively large, download time is much)

3, the log can be counted

A, to find the most frequently occurring anomalies, for tuning processing

B, Statistics crawler IP

C, Statistical user behavior, do cluster analysis, etc.

Based on the above requirements, I adopted the ELK (Elasticsearch + Logstash + kibana) of the scheme, installation methods please go to their official website: https://www.elastic.co/above, I mainly talk about the problems I encountered.

? ? ? ? ? ? 1, LVS distribution UDP request unsuccessful problem ? ? ? ? ? ?

In order not to affect the performance of the online cluster, we have taken a UDP way to transfer log messages, such as:

and load balancer I used LVS, when configuring LVS, I found that I need to use the Misc_check method in keepalived.conf to successfully detect real_server in PROTOCOL=UDP case, and the distribution is successful, The key configuration areas in keepalived.conf are:

Real_server Machine A 12201 {weight 1 Misc_check {Misc_path "/etc/keepalived/udp_check.sh machine a 1220 1 "Misc_timeout 10}}

And udp_check.sh This file is written by myself, the content is very simple:

/USR/BIN/NC-UZ-W1 | grep succeeded >/dev/nullexit $?

It is important to note that the permissions of this file are udp_check.sh, which I set here is 755

2, the Logstash cluster sends the log disorderly sequence

This problem solving method is very simple, unifies each logstash machine the system time can.

3, elasticsearch cluster brain fissure

There is a brain fissure, that cannot elect master, the solution is to increase heartbeat detection time, high load situation, the master response may appear slow, at this time can not be extreme think master down.

Elasticsearch + logstash + kibana build real-time log collection system "original"