Heka+flume+kafka+elk-Based logging system

Pre-Preparation

Elk Official Website: https://www.elastic.co/, package download and perfect documentation.

Zookeeper Official website: https://zookeeper.apache.org/

Kafka official website: http://kafka.apache.org/documentation.html, package download and perfect documentation.

Flume Official website: https://flume.apache.org/

Heka Official website: https://hekad.readthedocs.io/en/v0.10.0/

The system is a centos6.6,64 bit machine.

Version of the software used:

logstash:2.3.3

Jdk:1.8.0_25

elasticsearch:2.3.4

kibana:4.5.2

heka:0.10.0

flume:1.7.0

zookeeper:3.4.8

kafka:0.8.2.2

nginx:1.7.6

kafka-manager:1.3.1.6

Overall architecture Diagram

Installation Steps

Zero Installing JDK

A Install zookeeper cluster

Two Install Kafka cluster

Three Install Elasticsearch cluster

Four Install flume

Five Business Machine Installation Heka

Six Using Logstash docking Kafka and Elasticsearch clusters

Seven Install Kibana

Eight Install Nginx Reverse Proxy

Nine Install Kafka-manager

0. Installing JDK

Install JDK for all related machines, configure PATH, Class_path, Java_home

 

A Install zookeeper cluster

Ready to

The correspondence between hostname and IP

192.168.0.51 data1

192.168.0.49 data2

192.168.0.72 data3

You also need to configure the/etc/hosts file to add the above records

installation

Website download installation package, extract to/usr/local/public/, configuration, start "Other services similar, follow only to do configuration start instructions"

wget http://apache.fayea.com/zookeeper/zookeeper-3.4.8/zookeeper-3.4.8.tar.gz

cd/usr/local/public/

TAR-ZXVF zookeeper-3.4.8.tar.gz

Ln-s zookeeper-3.4.8 Zookeeper

configuration file

CD conf/

CP Zoo_samle.cfg Zoo.cfg

The content of Zoo.cfg

ticktime=2000

initlimit=10

Synclimit=5

Datadir=/data/zookeeper

clientport=2181

server.1=data1:2888:3888

server.2=data2:2888:3888

server.3=data3:2888:3888

Create the cluster flag file myID, which is the server in the configuration file. The following number.

echo "1" >/usr/local/public/zookeeper/data/myid

The other 2 machines are configured exactly the same, except that the contents of the myID file are 2 or 3.

Start Cluster

3 Servers executed sequentially: bin/zkserver.sh start

Test

bin/zkserver.sh Status View whether the current server belongs to leader or follower.

Bin/zkcli.sh-server gzhl-192-168-0-51.boyaa.com:2181 Connect to a zookeeper server.

Two Install Kafka cluster

installation

Similar to zookeeper, website download installation package, decompression.

configuration file Config/server.properties

Broker.id=1

Log.dirs=/disk1/bigdata/kafka

Zookeeper.connect=192.168.0.51:2181,192.168.0.49:2181,192.168.0.72:2181/kafka

The other 2 machines are configured exactly the same, except that the value of the Broker.id file is 2 or 3.

Start Cluster

3 Servers executed sequentially: nohup bin/kafka-server-start.sh config/server.properties 2>&1 >>/dev/null &

Test

bin/kafka-topics.sh--create--zookeeper gzhl-192-168-0-51.boyaa.com:2181,gzhl-192-168-0-49.boyaa.com:2181, Gzhl-192-168-2-147.boyaa.com:2181/kafka--topic Test//Create a topic

Three Install Elasticsearch cluster

installation

Similar to zookeeper, website download installation package, decompression.

configuration file Config/elasticsearch.yml

Cluster.name:dfqp-application

Node.name:node-2

Path.data:/disk1/es/,/disk2/es/,/disk3/es/,/disk4/es/,/disk5/es/,/disk6/es/,/disk7/es/,/disk8/es/,/disk9/es/,/ disk10/es/,/disk11/es/,/disk12/es/

Path.logs:/data/other/es/logs

network.host:192.168.0.150

Discovery.zen.ping.unicast.hosts: ["192.168.0.149", "192.168.0.150"]

The configuration of the 2 machines is exactly the same, except for the Node.name and Network.host values.

Start Cluster

Bin/elasticsearch-d

Test

Curl http://192.168.0.149:9200

Curl http://192.168.0.150:9200

Returns the following data indicating that the ES cluster is normal

{

"Name": "Node-2",

"Cluster_Name": "Dfqp-application",

"Version": {

"Number": "2.3.4",

"Build_hash": "e455fd0c13dceca8dbbdbb1665d068ae55dabe3f",

"Build_timestamp": "2016-06-30t11:24:31z",

"Build_snapshot": false,

"Lucene_version": "5.5.0"

},

"Tagline": "Your Know, for Search"

}

Four. Install flume

installation

Similar to zookeeper, website download installation package, decompression.

configuration file Conf/lremote.properties

Agent_remote.sources = S1

Agent_remote.sinks = K1

Agent_remote.channels = C1

Agent_remote.sources.s1.type = Thrift

Agent_remote.sources.s1.bind = 192.168.0.49

Agent_remote.sources.s1.port = 6666

Agent_remote.sources.s1.channels = C1

Agent_remote.sources.s1.threads = 20

Agent_remote.sinks.k1.type = Org.apache.flume.sink.kafka.KafkaSink

Agent_remote.sinks.k1.topic = Dfqp_server_log_from_heka

Agent_remote.sinks.k1.brokerList = 192.168.0.51:9092,192.168.0.49:9092,192.168.0.72:9092

Agent_remote.sinks.k1.requiredAcks = 0

agent_remote.sinks.k1.batchSize = 2000

Agent_remote.sinks.k1.channel = C1

agent_remote.sinks.k1.kafka.request.timeout.ms = 6000

agent_remote.sinks.k1.kafka.request.timeout.ms = 6000

Agent_remote.sources.s1.interceptors = I2

Agent_remote.sources.s1.interceptors.i2.type=org.apache.flume.sink.solr.morphline.uuidinterceptor$builder

Agent_remote.sources.s1.interceptors.i2.headername=key

Agent_remote.sources.s1.interceptors.i2.preserveexisting=false

Agent_remote.channels.c1.type = File

Agent_remote.channels.c1.capacity = 100000000

Agent_remote.channels.c1.dataDirs =/usr/local/public/flume/.flume_remote/c1/data_dir/

Agent_remote.channels.c1.checkpointDir =/usr/local/public/flume/.flume_remote/c1/checkpoint_dir/

Start Flume

Nohup bin/flume-ng agent-n agent_remote-c conf/-F Conf/remote.properties-dflume.log.file=flume_remote.log &

Five. Business Machine Installation Heka

installation

Website download source code, write Heka output plugin flume

configuration file Conf/log2flume.toml

[Hekad]

Maxprocs = 1

Base_dir = "./base_dir"

Log_info_filename = "Logs/info.log"

Log_error_filename = "Logs/error.log"

Log_file_max_size = 64

Log_file_max_backups = 7

[Logstreamerinput]

Log_directory = "/data/"

Journal_directory = "./base_dir"

File_match = ' (? p<dir1>[^/]*)/? (? p<dir2>[^/]*)/? (? p<dir3>[^/]*)/? (? p<dir4>[^/]*)/? (? p<dir5>[^/]*)/? (? p<dir6>[^/]*)/? (? p<dir7>[^/]*)/? (? p<file_name>.*) \.log\.? (? p<seq>\d*) '

Priority = ["^seq"]

Rescan_interval = "30s"

Oldest_duration = "1h"

differentiator = ["dfqp-", "Dir1", "--", "Dir2", "--", "Dir3", "--", "Dir4," "--", "DIR5", "--", "DIR6", "--", "DIR7", "--", "File_name", ". Log"]

[Paycenter_sqls_output]

Type = "Flumeoutput"

Message_matcher = "TRUE"

Address = "192.168.0.49:6666"

Encoder = "Payloadencoder"

batch_size = 2000

Use_buffering = False

[Payloadencoder]

Append_newlines = False

Start Hekad

./hekad-config CONF/LOG2FLUME.TOML

 

Six. Using Logstash docking Kafka and Elasticsearch clusters

installation

Similar to zookeeper, website download installation package, decompression.

configuration file etc/kafka_2_es.conf

Input {

Kafka {

topic_id => "My_kafka_topic"

group_id => "My_kafka_topic_group"

Consumer_threads => 4

Zk_connect => "Data1:2181,data2:2181,data3:2181/kafka"

Codec => "plain"

}

}

Output {

Elasticsearch {

Index => "server-log-%{+yyyy. MM.DD} "

Hosts => ["192.168.0.149", "192.168.0.150"]

}

}

The configuration of 2 machines is identical.

Start Logstash

Nohup bin/logstash-l logs/logstash${i}.log-f etc/kafka_2_es.conf >/dev/null &

Where ${i} is 1,2,3,4,5,6, namely a machine to start 6 logstash processes.

Seven. Install Kibana

installation

Similar to zookeeper, website download installation package, decompression.

configuration file Config/kibana.yml

Elasticsearch.url: "http://192.168.0.149:9200"

Start Logstash

Nohup./bin/kibana >/dev/null &

Eight Install Nginx Reverse proxy

configuration file/usr/local/nginx-1.7.6/conf/vhosts/vhost.conf

Server

{

Listen 80;

server_name log.oa.com;

Location/{

Proxy_pass Http://192.168.0.149:5601$request_uri;

Proxy_set_header Host $host;

Proxy_set_header X-real-ip $remote _addr;

Proxy_set_header x-forwarded-for $proxy _add_x_forwarded_for;

}

Location ~. *\. (gif|jpg|jpeg|png|bmp|swf|data|js|css|html|htm) $

{

Expires 365d;

}

Access_log/data/other/nginx_logs/${host}_${server_port}_access.log main;

}

Nine. Install Kafka-manager

git clone Https://github.com/yahoo/kafka-manager
CD Kafka-manager
./SBT Clean Dist

Will generate a Zip package, extract Kafka-manager, open the configuration file conf/application.conf, modify the ZK address.

Start: Nohup bin/kafka-manager-dconfig.file=conf/application.conf >/dev/null 2>&1 &

Visit: http://192.168.0.51:9000