In mission 800 operation and Maintenance summary of Haproxy---rsyslog----Kafka---Collector--es--kibana

This is my entire process of log analysis for haproxy in the unit.

We have been in the maintenance ES cluster configuration, and did not put a set of processes including the collection end of the code, all their own once, and the online collection of logs when we generally use the logstash, but the industry many people say logstash whether it is performance and stability is not very good, The advantage of Logstash is the simple configuration, this time I chose the Rsyslog

Today this haproxy log, I put the whole process to everyone go through, even if it is to let everyone understand the next

The specific process is as follows

Haproxy----LOCAL2 level----RSYSLOG----Kafka---collector (consumption Kafka, write to es)---es---kibana show

1.haproxy log By default requires a combination of rsyslog

Configuration files such as

local2.*/data1/logs/haproxy/haproxy.log

If we don't rsyslog, we'll have this situation with Logstash.

We still have to write the path of /data1/logs/haproxy/haproxy.log in input, so I think there's a big impact on performance .

But we used the rsyslog,rsyslog can directly from the LOCAL2 level directly split the log, this block I just for example, Rsyslog and logstash the difference is that rsyslog need plug-ins, and Rsyslog link Kafka need V8 version, and not yum install Rsyslog, need to compile the time to load module for Kafka

Prior to preparation, we need to enable Rsyslog to send data to Kafka first.

How to make Rsyslog support push to Kafka, steps below

/opt/test.sh

# # Install Rsyslog with Omkafka.

# # Omkafka enables Rsyslog to push logs to Kafka, a distributed message system.

# # See http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html

# # This installation the use of Yum to manage packages.

# # Add Rsyslog Repo

work_dir=$ (PWD)

Cd/etc/yum.repos.d

wget Http://rpms.adiscon.com/v8-stable/rsyslog.repo-O Rsyslog.repo

CD $WORK _dir

mkdir Rsyslog-install

CD Rsyslog-install

# Check Rsyslog version

# Rsyslog supports Kafka from v8.7.0

old_rsyslog_ver=$ (rsyslogd-version |head-n 1 | awk ' {print $} ')

# # Install Rsyslog dependency:libestr

Yum Install-y libestr-devel

# # Install Rsyslog Dependency:libee

Yum Install-y libee-devel

# # Install Rsyslog dependency:json-c

Yum Install-y json-c-devel

# # Install Rsyslog Denpendency:uuid

Yum Install-y libuuid-devel

# # Install Rsyslog Denpendency:liblogging-stdlog

Yum Install-y liblogging-devel

# # Install Rsyslog Denpendency:rst2man

Yum Install-y python-docutils

# # Install Librdkafka for Omkafka

wget Https://github.com/edenhill/librdkafka/archive/0.8.5.tar.gz-O librdkafka-0.8.5.tar.gz

Tar zxvf librdkafka-0.8.5.tar.gz

CD librdkafka-0.8.5

./configure

Make

Make install

Cd..

# # Install Rsyslog

wget Http://www.rsyslog.com/files/download/rsyslog/rsyslog-8.8.0.tar.gz-O rsyslog-8.8.0.tar.gz

Tar zxvf rsyslog-8.8.0.tar.gz

Export pkg_config_path=/usr/lib64/pkgconfig:/lib64/pkgconfig/

old_executable_path=$ (which rsyslogd)

executable_dir=$ (dirname "$old _executable_path")

CD rsyslog-8.8.0

./configure--prefix=/usr/local/rsyslog--sbindir= $executable _dir--libdir=/usr/lib64--enable-omkafka

Make

Make install

# # Show Installation Result:

new_rsyslog_ver=$ (rsyslogd-version |head-n 1 | awk ' {print $} ')

echo "Old Rsyslogd version:" $old _rsyslog_ver

echo "New rsyslogd version:" $new _rsyslog_ver

echo "Executable:" $ (which rsyslogd)

# # References:

# # Http://www.rsyslog.com/doc/master/installation/install_from_source.html

# # http://bigbo.github.io/pages/2015/01/21/syslog_kafka/

# # Http://blog.oldzee.com/?tag=rsyslog

# # http://www.rsyslog.com/newbie-guide-to-rsyslog/

# # Http://www.rsyslog.com/doc/master/configuration/modules/omkafka.html

2.CP./rsyslog-install/librdkafka-0.8.5/src/librdkafka.so.1/lib64/

chmod 755/lib64/librdkafka.so.1

3.CP./rsyslog-8.8.0/plugins/omkafka/.libs/omkafka.so/lib64/rsyslog/

chmod 755/lib64/rsyslog/omkafka.so

4.rsyslogd-n Test rsyslog configuration file is correct

5./lib64/rsyslog/below are the modules that Rsyslog can load

OK, now we can get Rsyslog to Kafka.

We're going to define the data in advance to the fields in the ES index, and we'll process the logs based on these fields.

Rsyslog for haproxy log processing, of course, I post the processing method below is not the best, Rsyslog also support Logstash filter Grok has its own similar plug-in, with Mmnormalize plug-in more efficient

650) this.width=650; "Src=" http://set2.exmail.qq.com/cgi-bin/viewfile?f= dfaf6c355be1606a4a0fe026c5e48770d0b1cffbc6174cc4830fc741f3e99d1013efa16c8a1ceb7edcf6c97be9079c85866f27cfadec6d673faad9705 12114170b00bf6470e1142d7bf60cef9bdf36a8ff2d9ff7f5f84c916014643793fbd7f0&usewmcache=1&sid= vyhifsbskcmt00gf,7 "style=" border:none;vertical-align:middle; "alt=" viewfile?f=dfaf6c355be1606a4a0fe026c5e48 "/ >

We've now done the right split for the logs we need to get to ES.

We're going to Kafka the data in the next room.

local2.* Action (type= "Omkafka" broker= "[ 172.16.10.130:9092,172.16.10.131:9092,172.16.10.132:9092,172.16.10.139:9092,172.16.10.140:9092] "topic=" EAGLEYE _log_haproxy_channel "partitions.number=", "confparam=[", "Compression.codec=snappy", "s

Ocket.keepalive.enable=true "] queue.saveonshutdown=" on "queue.size=" 10000000 "queue.type=" LinkedList " queue.highwatermark= "600000" queue.lowwatermark= "20000" queue.discardmark= "800000" queue.maxfilesize= "1g" Queue.maxdiskspace= "10g" acti

On. Resumeinterval= "Ten" action. Resumeretrycount= "-1" action.reportsuspension= "on" action.reportsuspensioncontinuation= "on" template= "Json_lines" )

The specific parameters can be viewed on the official website.

Let's go see if Kafka's topic Eagleye_log_haproxy_channel has data coming up.

650) this.width=650; "Src=" http://set2.exmail.qq.com/cgi-bin/viewfile?f= Dfaf6c355be1606a4a0fe026c5e48770d0b1cffbc6174cc4830fc741f3e99d1013efa16c8a1ceb7edcf6c97be9079c85866f27cfadec6d67e710fda54 7fba17f5ece28939338a822f94669847e6499846d453087ce2b66877f6a9f17eb1c0e88&usewmcache=1&sid= vyhifsbskcmt00gf,7 "style=" border:none;vertical-align:middle; "alt=" viewfile?f=dfaf6c355be1606a4a0fe026c5e48 "/ >

You can see that the data has been written in Kafka.

Now we're going to write the code for the collection end.

When it comes to the collection end, it's actually the consumer and inserting it into ES.

Code, I'm not going to post it.

The approximate logic is to start with 2 separate threads a thread is responsible for consuming Kafka, a thread calls ES API or the _bulk method of the RESTful interface to bulk insert JSON data

Let's see if ES already has data.

650) this.width=650; "Src=" http://set2.exmail.qq.com/cgi-bin/viewfile?f= dfaf6c355be1606a4a0fe026c5e48770d0b1cffbc6174cc4830fc741f3e99d1013efa16c8a1ceb7edcf6c97be9079c85866f27cfadec6d67cffa90367 23a06020d54139057cc7602199cd89faf2ccd55c84ce4a63b1350262623f3fd0068eecd&usewmcache=1&sid= vyhifsbskcmt00gf,7 "style=" border:none;vertical-align:middle; "alt=" viewfile?f=dfaf6c355be1606a4a0fe026c5e48 "/ >

OK now there's data in the ES cluster.

Now let's go to Kibana and add the chart.

How to add I'm not here to say, let's show you some pictures

Kibana I don't show it because there's most grateful information.

Can see the total number of requests the average response time has those IP links haproxy haproxy call those IP those requests are not normal can be seen through the chart, there are some requirements I will add to the Kibana in the future

So the whole process went down, we are now the majority of others to give us the demand, we just go to configure, but we can give ourselves some of the needs, we come to analyze some of the log information we think useful, haproxy log analysis, I think is an example for everyone to reference, thank you

This article from "Expect volume synchronization data" blog, declined reprint!

In mission 800 operation and Maintenance summary of Haproxy---rsyslog----Kafka---Collector--es--kibana