Elk Analysis Nginx Access and error logs _elk

1 nginx Log Format configuration

[Root@elk-5-10 config]# cd/usr/local/nginx/conf/
[Root@elk-5-10 conf]# VI nginx.conf

Log_format access ' $http _host $remote _addr-$remote _user [$time _local] "$request"
' $status $body _bytes_sent ' $http _referer '
' $http _user_agent ' $http _x_forwarded_for ';

2nd log Format Data samples

2.1 Access log:

Ss00.xxxxxx.me 150.138.154.157--[25/jul/2017:03:02:35 +0800] "get/csm/7_527.html http/1.1" 304 0 "http://www.twww.com /tetris/page/64000159042/?ad_id=62928537191&cid=62928889880&req_id=0 "" Mozilla/5.0 (Linux; Android 6.0.1; Redmi 4X build/mmb29m; WV) applewebkit/537.36 (khtml, like Gecko) version/4.0 chrome/55.0.2883.91 Mobile safari/537.36 "183.204.183.118

2.2 Error Log

2017/07/25 15:23:29 [ERROR] 24881#0: *33 open () "/usr/local/nginx-1.12.0/html/favicon.ico" failed (2:no such file or dire Ctory), client:192.168.1.103, server:www.zyb.com, Request: "Get/favicon.ico http/1.1", Host: "Www.zyb.com"

3 Logstash configuration file

Input {
File {
Type => "Nginx-access"
Path => "/data/weixin.sys.mingyaohui.com.log"
Start_position => Beginning
}
File {
Type => "Nginx-error"
Path => "/data/nginx_error.log"
Start_position => Beginning
}
}

Filter {
if [type] = = "Nginx-access" {
Grok {
Match => ["message", "%{iporhost:clientip}%{nguser:ident}%{nguser:auth} \[%{httpdate:timestamp}\]"%{WORD:verb}% {Uripathparam:request} Http/%{number:httpversion} "%{number:response} (?:%{number:bytes}|-) (?:" (?:%{uri:referrer}|-) "|%{QS:referrer}"%{ Qs:agent}%{iporhost:forwordip} "]}
}
else if [type] = = "Nginx-error" {
Grok {
Match => ["message", "(? <timestamp>%{year}[./-]%{monthnum}[./-]%{monthday}[-]%{time}) \[%{loglevel: Severity}\]%{posint:pid}#%{number}:%{greedydata:errormessage} (?:, Client: (? <clientip>%{ip}|%{hostname})) ( ?:, Server:%{iporhost:server}?) (?:, Request:%{qs:request})? (?:, Upstream: (? <upstream>\ "%{uri}\" |%{qs}))? (?:, Host:%{qs:request_host})? (?:, referrer: \ "%{uri:referrer}\")? "]
}
}

# Add Geo-location Info
GeoIP {
SOURCE => "ClientIP"
}
}

Output {
Elasticsearch {
Hosts => ["10.0.0.10"]
Index => "%{type}-%{+yyyy. MM.DD} "
}
}

4 Kibana Analysis Effect chart

Resources

https://grokdebug.herokuapp.com/patterns#

Https://github.com/adventure-yunfei/ELK-for-nginx-log/blob/master/logstash.conf