Logstash analysis Nginx, DNS log

ElkAnalysisNginx,DnsLog

Deployment environment

192.168.122.187	Logstash-1.5.1 elasticsearch-1.6.0 kibana-4.1.1	Centos6.4
192.168.122.1	redis-2.8	centos7.1
192.168.122.2	nginx logstash-1.5.2 supervisor-2.1-9 java-1.7	centos6.4
192.168.122.247	BIND9 logstash-1.5.2 supervisor-2.1-9 java-1.7	Centos6.2

the installation process is no longer described, refer to http://kibana.logstash.es/content/logstash/get_start/install.html

Several places to note during installation

1, Java is best 1.7

2, the server on the logstash I directly with the RPM can be used, but the agent side is not so, did not delve

3, Elasticsearch,kibana and agent side of the Logstash I have to run with the supervisor

4, Supervisor directly is the epel yum pack

Paste the configuration192.168.122.187on: Logstash the configuration

Server End of Logstash is a rpm installation of

[Email protected] ~]# cat/etc/logstash/conf.d/central.conf

Input {

Redis {

Host = "192.168.122.1"

Port = 6379

Type = "Redis-input"

data_type = "List"

Key = "Logstash"

codec = ' json '

}

}

Output {

Elasticsearch {

Host = "127.0.0.1"

}

}

Elasticsearch

/USR/LOCAL/ELASTICSEARCH-1.6.0/CONFIG/ELASTICSEARCH.YML Keep the default

Kibana

/USR/LOCAL/KIBANA-4.1.1-LINUX-X64/CONFIG/KIBANA.YML Keep the default

192.168.122.1on

The Redis configuration is not moving ...

192.168.122.2onNginxof the

#nginx这里的区别就是log这块的配置, formatted as a JSON

Log_format json ' {"@timestamp": "$time _iso8601", '

' Host ': ' $server _addr ', '

' "ClientIP": "$remote _addr", '

' Size ': $body _bytes_sent, '

' "ResponseTime": $request _time, '

' "Upstreamtime": "$upstream _response_time", '

' "Upstreamhost": "$upstream _addr", '

' "Http_host": "$host", '

' URL ': ' $uri ', '

' "Xff": "$http _x_forwarded_for", '

' "Referer": "$http _referer", '

"Agent": "$http _user_agent", '

' Status ': ' $status '} ';

-----------------------------

Access_log/var/log/nginx/zabbix_access.log JSON;

Logstashof the

[Email protected] ~]# cat/usr/local/logstash-1.5.2/conf/shipper.conf

Input {

File {

Type = "Test-nginx"

Path = ["/var/log/nginx/zabbix_access.log"]

codec = "JSON"

}

}

Output {

StdOut {}

Redis {

Host = "192.168.122.1"

Port = 6379

data_type = "List"

Key = "Logstash"

}

}

Supervisorof the

[[email protected] ~]# cat/etc/supervisord.conf |grep-v \;

[Supervisord]

[Program:logstash]

Command=/usr/local/logstash-1.5.2/bin/logstash Agent--verbose--config/usr/local/logstash-1.5.2/conf/ Shipper.conf--log/usr/local/logstash-1.5.2/logs/stdout.log

process_name=% (program_name) s

Numprocs=1

Autostart=true

Autorestart=true

Startretries=5

Exitcodes=0

Stopsignal=kill

Stopwaitsecs=5

Redirect_stderr=true

[Supervisorctl]

192.168.122.247onBindis configured with the defaultLogstashof the

[Email protected] ~]# cat/usr/local/logstash/conf/shipper.conf

Input {

File {

Type = "Dnslog"

Path = ["/home/dnslog/*.log"]

}

}

Filter {

#由于dns日志没办法定义成json, I'm not grok, so I cut it with mutate.

Mutate {

Gsub = ["Message", "#", "" "]

split = ["Message", "" "]

}

Mutate {

Add_field = {

"Client" = "%{[message][5]}"

"Domain_Name" = "%{[message][10]}"

"Server" = "%{[message][14]}"

}

}

}

Output {

StdOut {}

Redis {

Host = "192.168.122.1"

Port = 6379

data_type = "List"

Key = "Logstash"

}

}

Supervisorof the

[Email protected] ~]# cat/etc/supervisord.conf |grep-v \;|grep-v ^$

[Supervisord]

[Supervisorctl]

[Program:logstash]

Command=/usr/local/logstash/bin/logstash Agent--verbose--config/usr/local/logstash/conf/shipper.conf--log/usr/ Local/logstash/logs/stdout.log

process_name=% (program_name) s

Numprocs=1

Autostart=true

Autorestart=true

Startretries=5

Exitcodes=0

Stopsignal=kill

Stopwaitsecs=5

Redirect_stderr=true

ConfigurationKibananginx

1, in discover search Nginx related logs, and then save

2. Deploy a single chart in visualize, then save

3, in the dashboard will be a few Nginx visualize chart linked together

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/70/FE/wKioL1XDEQOBCHTsAAZID6sFRdY913.jpg "style=" float: none; "title=" Qq20150806143006.png "alt=" Wkiol1xdeqobchtsaazid6sfrdy913.jpg "/>

Dns

1. Search DNS -related logs in Discover, then save

2. Deploy a single chart in visualize, then save

3. Connect several DNS visualize charts in dashboard

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/70/FE/wKioL1XDESeQlPt1AATGBsUM800144.jpg "title=" Qq20150806143238.png "alt=" Wkiol1xdeseqlpt1aatgbsum800144.jpg "/>

problems encounteredCustom-DefinedFieldin theDiscovercan see, but in the makingVisualizewhen you don't see

This situation is caused by a field that does not have the index refreshed , the default index is logstash-*, and is seenin the "Settings"-indices , click logstash-* go in, click the Refresh button

This article is from the "My Life for My Wife" blog, so be sure to keep this source http://shanks.blog.51cto.com/3899909/1682273

Logstash analysis Nginx, DNS log