ElkAnalysisNginx,DnsLog
Deployment environment
192.168.122.187 |
Logstash-1.5.1 elasticsearch-1.6.0 kibana-4.1.1 |
Centos6.4 |
192.168.122.1 |
redis-2.8 |
centos7.1 |
192.168.122.2 |
nginx logstash-1.5.2 supervisor-2.1-9 java-1.7 |
centos6.4 |
192.168.122.247 |
BIND9 logstash-1.5.2 supervisor-2.1-9 java-1.7 |
Centos6.2 |
the installation process is no longer described, refer to http://kibana.logstash.es/content/logstash/get_start/install.html
Several places to note during installation
1, Java is best 1.7
2, the server on the logstash I directly with the RPM can be used, but the agent side is not so, did not delve
3, Elasticsearch,kibana and agent side of the Logstash I have to run with the supervisor
4, Supervisor directly is the epel yum pack
Paste the configuration192.168.122.187on:
Logstash
the configuration
Server End of Logstash is a rpm installation of
[Email protected] ~]# cat/etc/logstash/conf.d/central.conf
Input {
Redis {
Host = "192.168.122.1"
Port = 6379
Type = "Redis-input"
data_type = "List"
Key = "Logstash"
codec = ' json '
}
}
Output {
Elasticsearch {
Host = "127.0.0.1"
}
}
Elasticsearch
/USR/LOCAL/ELASTICSEARCH-1.6.0/CONFIG/ELASTICSEARCH.YML Keep the default
Kibana
/USR/LOCAL/KIBANA-4.1.1-LINUX-X64/CONFIG/KIBANA.YML Keep the default
192.168.122.1on
The Redis configuration is not moving ...
192.168.122.2onNginxof the
#nginx这里的区别就是log这块的配置, formatted as a JSON
Log_format json ' {"@timestamp": "$time _iso8601", '
' Host ': ' $server _addr ', '
' "ClientIP": "$remote _addr", '
' Size ': $body _bytes_sent, '
' "ResponseTime": $request _time, '
' "Upstreamtime": "$upstream _response_time", '
' "Upstreamhost": "$upstream _addr", '
' "Http_host": "$host", '
' URL ': ' $uri ', '
' "Xff": "$http _x_forwarded_for", '
' "Referer": "$http _referer", '
"Agent": "$http _user_agent", '
' Status ': ' $status '} ';
-----------------------------
Access_log/var/log/nginx/zabbix_access.log JSON;
Logstashof the
[Email protected] ~]# cat/usr/local/logstash-1.5.2/conf/shipper.conf
Input {
File {
Type = "Test-nginx"
Path = ["/var/log/nginx/zabbix_access.log"]
codec = "JSON"
}
}
Output {
StdOut {}
Redis {
Host = "192.168.122.1"
Port = 6379
data_type = "List"
Key = "Logstash"
}
}
Supervisorof the
[[email protected] ~]# cat/etc/supervisord.conf |grep-v \;
[Supervisord]
[Program:logstash]
Command=/usr/local/logstash-1.5.2/bin/logstash Agent--verbose--config/usr/local/logstash-1.5.2/conf/ Shipper.conf--log/usr/local/logstash-1.5.2/logs/stdout.log
process_name=% (program_name) s
Numprocs=1
Autostart=true
Autorestart=true
Startretries=5
Exitcodes=0
Stopsignal=kill
Stopwaitsecs=5
Redirect_stderr=true
[Supervisorctl]
192.168.122.247onBindis configured with the defaultLogstashof the
[Email protected] ~]# cat/usr/local/logstash/conf/shipper.conf
Input {
File {
Type = "Dnslog"
Path = ["/home/dnslog/*.log"]
}
}
Filter {
#由于dns日志没办法定义成json, I'm not grok, so I cut it with mutate.
Mutate {
Gsub = ["Message", "#", "" "]
split = ["Message", "" "]
}
Mutate {
Add_field = {
"Client" = "%{[message][5]}"
"Domain_Name" = "%{[message][10]}"
"Server" = "%{[message][14]}"
}
}
}
Output {
StdOut {}
Redis {
Host = "192.168.122.1"
Port = 6379
data_type = "List"
Key = "Logstash"
}
}
Supervisorof the
[Email protected] ~]# cat/etc/supervisord.conf |grep-v \;|grep-v ^$
[Supervisord]
[Supervisorctl]
[Program:logstash]
Command=/usr/local/logstash/bin/logstash Agent--verbose--config/usr/local/logstash/conf/shipper.conf--log/usr/ Local/logstash/logs/stdout.log
process_name=% (program_name) s
Numprocs=1
Autostart=true
Autorestart=true
Startretries=5
Exitcodes=0
Stopsignal=kill
Stopwaitsecs=5
Redirect_stderr=true
ConfigurationKibananginx
1, in discover search Nginx related logs, and then save
2. Deploy a single chart in visualize, then save
3, in the dashboard will be a few Nginx visualize chart linked together
650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/70/FE/wKioL1XDEQOBCHTsAAZID6sFRdY913.jpg "style=" float: none; "title=" Qq20150806143006.png "alt=" Wkiol1xdeqobchtsaazid6sfrdy913.jpg "/>
Dns
1. Search DNS -related logs in Discover, then save
2. Deploy a single chart in visualize, then save
3. Connect several DNS visualize charts in dashboard
650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/70/FE/wKioL1XDESeQlPt1AATGBsUM800144.jpg "title=" Qq20150806143238.png "alt=" Wkiol1xdeseqlpt1aatgbsum800144.jpg "/>
problems encounteredCustom-DefinedFieldin theDiscovercan see, but in the makingVisualizewhen you don't see
This situation is caused by a field that does not have the index refreshed , the default index is logstash-*, and is seenin the "Settings"-indices , click logstash-* go in, click the Refresh button
This article is from the "My Life for My Wife" blog, so be sure to keep this source http://shanks.blog.51cto.com/3899909/1682273
Logstash analysis Nginx, DNS log