Logstash learn a little mind

Logstash learn a little mind

tags (space delimited): Log collection

Introduce

Logstash is a tool for managing events and logs. You can use it to collect logs, parse them, and store them for
Later use (like, for searching). –http://logstash.net

Since 2013 Logstash was acquired by ES Company, ELK Stask officially known as the official language, many companies are beginning to ELK practice, we are no exception, how to use Sina Analytics processing 3.2 billion real-time logs? A picture of

This is a more common architecture:
(1) Kafka: Receive Message Queuing for user logs.
(2) Logstash: Do log parsing, unified into JSON output to elasticsearch.
(3) Elasticsearch: The core technology of real-time log Analysis service, a schemaless, real-time data storage service, through index organization data, both powerful search and statistical functions.
(4) Kibana: Data visualization component based on Elasticsearch, strong data visualization capability is an important reason for many companies to choose Elk Stack.

But many log collection framwork, like Flume,scribe,fluent, why choose Logstash?

The reason is simple:

1. Deployment startup is easy, just need to have a JDK on the OK
2. Simple configuration, no coding required

3. A regular expression that supports the collection of log paths, unlike Flume, which must write dead file names to collect, Logstash not, like this

   Path = ["/var/log/. Log"]

   There's a flume vs Fluentd vs Logstash can see

Logstash Examples

Logstash event Processing Flow atmosphere three stages:input, Filter,output. Input supports a lot, such as File,redis,kafka, and so on, filter is mainly on the log of input to do its own processing, output is to the third party framework that you want to store log, such as Kafka,redis, Elasticsearch,db or something, the specific view of the official website.
Nonsense not much to say, the beginning example:
1. The most simple example
Input and output are standard input and output

[joeywen@192 logstash]$ ‘input { stdin { } } output { stdout {}}‘Logstash startup completed>hello world  ## 输入的内容>2015-08-02T05:26:55.564Z joeywens-MacBook-Pro.local hello world   ## logstash收集的内容

1. Writing config file

input {  file {    path => ["/var/log/*.log"]    "syslog"    codec => multiline {      "(^\d+\serror)|(^.+Exception:.+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Causedby:.+)"      "previous"    }  }}output {  stdout {  codec => rubydebug }  #  elasticsearch {  #      host => ‘localhost‘  #      protocol => ‘transport‘  #      cluster => ‘elasticsearch‘#       index => ‘logstash-joeymac-%{+YYYY.MM.dd}‘#   }}

The input is file form, collect the system log, if an exception occurs, usually the exception will be more rows, here with codec = Multiline to the occurrence of the exception of the multi-line conversion to a line of input

The output is ES, or you can open the stdout as a debug to see what the output is. Run the command as well as the output

[[Email protected]192logstash]$ bin/logstash-f Sys.conflogstash Startup completed{"@timestamp"="2015-08-02t05:36:08.972z","Message"="2 13:36:08 joeywens-macbook-pro.local googlesoftwareupdateagent[1976]: 2015-08-02 13:34:51.764 GOOGLESOFTWAREUPDATEAGENT[1976/0XB029B000] [lvl=2]-[ksupdateengine (privatemethods) updateFinish] KSUpdateEngine Update processing complete. ","@version"="1","Host"="Joeywens-macbook-pro.local","Path"="/var/log/system.log","Type"="Syslog"}{"@timestamp"="2015-08-02t05:36:08.973z","Message"="2 13:36:08 joeywens-macbook-pro.local googlesoftwareupdateagent[1976]: 2015-08-02 13:36:08.105 GOOGLESOFTWAREUPDATEAGENT[1976/0XB029B000] [lvl=3]-[ksagentuploader Fetcher:failedwitherror:] Failed to upload stats To <nsmutableurlrequest https://tools.google.com/service/update2> with error error Domain=nsurlerrordomain Code =-1001 \ "The Request timed out.\" USERINFO=0X3605F0 {nserrorfailingurlstringkey=https://tools.google.com/service/ Update2, _kcfstreamerrorcodekey=60, Nserrorfailingurlkey=https://tools.google.com/service/update2, Nslocalizeddescription=the request timed out., _kcfstreamerrordomainkey=1, nsunderlyingerror=0x35fd30 \ "The request Timed out.\ "}","@version"="1","Host"="Joeywens-macbook-pro.local","Path"="/var/log/system.log","Type"="Syslog"}{"@timestamp"="2015-08-02t05:36:08.973z","Message"="2 13:36:08 joeywens-macbook-pro.local googlesoftwareupdateagent[1976]: 2015-08-02 13:36:08.272 GOOGLESOFTWAREUPDATEAGENT[1976/0XB029B000] [lvl=3]-[ksagentapp uploadstats:] Failed to upload stats < Ksstatscollection:0x4323e0 path=\ "/users/joeywen/library/google/googlesoftwareupdate/stats/keystone.stats\", Count=6, stats={","@version"="1","Host"="Joeywens-macbook-pro.local","Path"="/var/log/system.log","Type"="Syslog"}

What do I do if I want to add or remove a field? Filter, it's time to go.

1. Filter
   Filter is very powerful, can make any changes to the content of input, input content will be converted to a map called event, which holds the key/value, as you see the output, @timestamp, type, @version, Host,message and so on, are all key in the event, you can start Ruby programming plugin to make any changes in the filter
   Such as:

input {file {path =>  ["/var/log/*.log "] type =>  " syslog "  codec =>  Multiline {pattern =>  what =>  " previous " }}}filter {if  [Type] =~ /^syslog/ { Ruby {Code =>   "file_name = event[' path ']. Split ('/') [-1] event[' file_name '] = file_name "}}}output {stdout {codec =>  Rubydebug}} 

I made changes to the event that the type has started with syslog, calling Ruby programming
to see the output

[[email protected]192  logstash]$ bin/logstash-f Sys.conflogstash startup completed{ "@timestamp"  = =  " 2015-08-02t05:46:52.771z ", " message " = ,  "@version"  = = ,  "host" /span> =  "joeywens-macbook-pro.local" ,  "path" =  "/var/log/system.log" ,  "type"  =  "syslog" ,  "file_name"  + " System.log "} 

You can see a more file_name field,
If the relative message is parsed, you need to call Grok plugin to do it, Grok is a very powerful plugin, such as

input {  file {    "/var/log/http.log"  }}filter {  grok {    patterns_dir => ["/opt/logstash/patterns""/opt/logstash/extra_patterns"]    "message""%{IP:client} %{WORD:method} %{URIPATHPARAM:request} %{NUMBER:bytes} %{NUMBER:duration}" }  }}

A regular match is called for the message field, and the syntax is%{syntax:semantic}
The first syntax is the regular expression name, the second is the name of the field that matches successfully, and these syntax exist in the specified Pattern_dir directory, in the following format:

NAME PATTERN
As number \d+

You can also use mutate to make changes to the most event key and value, including Remove,add,update,rename, and so on, to see (Logstash document) [Https://www.elastic.co/guide /en/logstash/current/plugins-filters-mutate.html]

Here's a concrete example.
Configuration:

Input {file {path=["/var/log/*.log"] Type= "Syslog"Codec=Multiline {pattern= "(^\d+\serror) | (^.+exception:.+) | (^\s+at. +) | (^\s+ ... \d+ more) | (^\s*causedby:.+) "What= "Previous"}}}filter {if[Type] =~/^syslog/{Ruby {code= "file_name = event[' path '].split ('/') [-1] event[' file_name '] = file_name"} grok {Patterns_dir=["./patterns/*"] Match={"Message"= "%{mac_book:joeymac}"}} mutate {rename={"file_name"= "FileName"} Add_field={"Foo_%{joeymac}"= "Hello World, from%{host}"}}}}output {stdout {codec=Rubydebug}}

Output

[[Email protected]192logstash]$ bin/logstash-f Sys.conflogstash Startup completed{"@timestamp"="2015-08-02t06:10:13.161z","Message"="2 14:10:12 joeywens-macbook-pro com.apple.xpc.launchd[1] (com.apple.quicklook[2206]): Endpoint has been Activated through Legacy launch (3) APIs. Please switch to XPC or bootstrap_check_in (): Com.apple.quicklook ","@version"="1","Host"="Joeywens-macbook-pro.local","Path"="/var/log/system.log","Type"="Syslog","Joeymac"="Joeywens-macbook-pro","FileName"="System.log","Foo_joeywens-macbook-pro"="Hello World, from Joeywens-macbook-pro.local"}

Reprint please indicate the source

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

Logstash learn a little mind