Logstash analysis Nginx, DNS log

Source: Internet
Author: User
Tags kibana logstash

ElkAnalysisNginx,DnsLog

Deployment environment

192.168.122.187

Logstash-1.5.1 elasticsearch-1.6.0 kibana-4.1.1

Centos6.4

192.168.122.1

redis-2.8

centos7.1

192.168.122.2

nginx logstash-1.5.2 supervisor-2.1-9 java-1.7

centos6.4

192.168.122.247

BIND9 logstash-1.5.2 supervisor-2.1-9 java-1.7

Centos6.2

the installation process is no longer described, refer to http://kibana.logstash.es/content/logstash/get_start/install.html

Several places to note during installation

1, Java is best 1.7

2, the server on the logstash I directly with the RPM can be used, but the agent side is not so, did not delve

3, Elasticsearch,kibana and agent side of the Logstash I have to run with the supervisor

4, Supervisor directly is the epel yum pack

Paste the configuration192.168.122.187on: Logstash the configuration

Server End of Logstash is a rpm installation of

[Email protected] ~]# cat/etc/logstash/conf.d/central.conf
Input {
Redis {
Host = "192.168.122.1"
Port = 6379
Type = "Redis-input"
data_type = "List"
Key = "Logstash"
codec = ' json '
}
}

Output {
Elasticsearch {
Host = "127.0.0.1"
}
}
Elasticsearch
/USR/LOCAL/ELASTICSEARCH-1.6.0/CONFIG/ELASTICSEARCH.YML Keep the default
Kibana
/USR/LOCAL/KIBANA-4.1.1-LINUX-X64/CONFIG/KIBANA.YML Keep the default
192.168.122.1on
The Redis configuration is not moving ...
192.168.122.2onNginxof the
#nginx这里的区别就是log这块的配置, formatted as a JSON
Log_format json ' {"@timestamp": "$time _iso8601", '
' Host ': ' $server _addr ', '
' "ClientIP": "$remote _addr", '
' Size ': $body _bytes_sent, '
' "ResponseTime": $request _time, '
' "Upstreamtime": "$upstream _response_time", '
' "Upstreamhost": "$upstream _addr", '
' "Http_host": "$host", '
' URL ': ' $uri ', '
' "Xff": "$http _x_forwarded_for", '
' "Referer": "$http _referer", '
"Agent": "$http _user_agent", '
' Status ': ' $status '} ';
-----------------------------
Access_log/var/log/nginx/zabbix_access.log JSON;
Logstashof the
[Email protected] ~]# cat/usr/local/logstash-1.5.2/conf/shipper.conf
Input {
File {
Type = "Test-nginx"
Path = ["/var/log/nginx/zabbix_access.log"]
codec = "JSON"
}
}

Output {
StdOut {}
Redis {
Host = "192.168.122.1"
Port = 6379
data_type = "List"
Key = "Logstash"
}
}
Supervisorof the
[[email protected] ~]# cat/etc/supervisord.conf |grep-v \;
[Supervisord]
[Program:logstash]
Command=/usr/local/logstash-1.5.2/bin/logstash Agent--verbose--config/usr/local/logstash-1.5.2/conf/ Shipper.conf--log/usr/local/logstash-1.5.2/logs/stdout.log
process_name=% (program_name) s
Numprocs=1
Autostart=true
Autorestart=true
Startretries=5
Exitcodes=0
Stopsignal=kill
Stopwaitsecs=5
Redirect_stderr=true
[Supervisorctl]
192.168.122.247onBindis configured with the defaultLogstashof the
[Email protected] ~]# cat/usr/local/logstash/conf/shipper.conf
Input {
File {
Type = "Dnslog"
Path = ["/home/dnslog/*.log"]
}
}
Filter {
#由于dns日志没办法定义成json, I'm not grok, so I cut it with mutate.
Mutate {
Gsub = ["Message", "#", "" "]
split = ["Message", "" "]
}
Mutate {
Add_field = {
"Client" = "%{[message][5]}"
"Domain_Name" = "%{[message][10]}"
"Server" = "%{[message][14]}"
}
}
}
Output {
StdOut {}
Redis {
Host = "192.168.122.1"
Port = 6379
data_type = "List"
Key = "Logstash"
}
}
Supervisorof the
[Email protected] ~]# cat/etc/supervisord.conf |grep-v \;|grep-v ^$
[Supervisord]
[Supervisorctl]
[Program:logstash]
Command=/usr/local/logstash/bin/logstash Agent--verbose--config/usr/local/logstash/conf/shipper.conf--log/usr/ Local/logstash/logs/stdout.log
process_name=% (program_name) s
Numprocs=1
Autostart=true
Autorestart=true
Startretries=5
Exitcodes=0
Stopsignal=kill
Stopwaitsecs=5
Redirect_stderr=true
ConfigurationKibananginx

1, in discover search Nginx related logs, and then save

2. Deploy a single chart in visualize, then save

3, in the dashboard will be a few Nginx visualize chart linked together

650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M00/70/FE/wKioL1XDEQOBCHTsAAZID6sFRdY913.jpg "style=" float: none; "title=" Qq20150806143006.png "alt=" Wkiol1xdeqobchtsaazid6sfrdy913.jpg "/>

Dns

1. Search DNS -related logs in Discover, then save

2. Deploy a single chart in visualize, then save

3. Connect several DNS visualize charts in dashboard

650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/70/FE/wKioL1XDESeQlPt1AATGBsUM800144.jpg "title=" Qq20150806143238.png "alt=" Wkiol1xdeseqlpt1aatgbsum800144.jpg "/>

problems encounteredCustom-DefinedFieldin theDiscovercan see, but in the makingVisualizewhen you don't see

This situation is caused by a field that does not have the index refreshed , the default index is logstash-*, and is seenin the "Settings"-indices , click logstash-* go in, click the Refresh button


This article is from the "My Life for My Wife" blog, so be sure to keep this source http://shanks.blog.51cto.com/3899909/1682273

Logstash analysis Nginx, DNS log

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.