"Logstash"-Logstash Configuration Language Basics

It's hard to find logstash Chinese material on the internet, Ruby didn't know it, it was too difficult to read official documents, and my requirements are not high, using Loggstash can extract the desired fields.

The following is purely understandable:

Logstash Configuration Format

#官方文档: http://www.logstash.net/docs/1.4.2/input {... #读取数据, Logstash has provided very many plugins, such as the ability to read data from file, Redis, Syslog, and so on}filter { ... #想要从不规则的日志中提取关注的数据, it needs to be handled here. Commonly used Grok, mutate and other}output {... #输出数据, data output to file, Elasticsearch, etc.} after processing

Logstash Processing Process:

1. Read data from the plug-in in input, row-by-line (as with awk)

file{

Path = "/var/log/maillog"

Start_position = "Beginning"

}

2. Data processing in the filter

First, read the first line and pass the content to the Message field (the message is similar to the one in awk).

grok{} takes the required data from the message and uses the regular expression primarily.

mutate{} mainly modifies data, such as obtaining a value for a field that can be processed using mutate.

3. Export the processed data to each plugin

After processing a row of data, repeat the above action until the data is fully processed.

Logstash Configuration language

Website: http://www.logstash.net/docs/1.4.2/configuration

#: Comment boolean:true  or falseexamples:debug => truestring (string) name =>  "Hello  World "#字符串放在双引号内abc  => "%{name} "#这样abc的值就是name的值Numberport  => 33array (Array) path = > [  "/var/log/messages",  "/var/log/*.log"  ]path =>  "/data/mysql/mysql.log" #path包含三个路径. hashmatch => {   "field1"  =>  "value1"    "Field2"  =>  " Value2 " &nbsp ...} #把多个字段放在 {}, each field uses   "key"  =>  "Value" field references (field Reference) {   "Agent":  " mozilla/5.0  (compatible; msie 9.0) ",  " IP ": " 192.168.24.44 ",  " request ": "/index.html "  " Response ": {    " status ": 200,      "bytes": 52353  },   "ua": {     "OS":  " Windows 7 "&NBSP;&NBSP;}} #字段引用使用 [], such as using status to judge, if [status] = 200 {} #若是要Get the value of the field, using  %{ip} #取os的值, you need this: [Ua][os], you can think of UA as the array name, OS is subscript. conditionals (conditional statement) if expression { &nbsp ...} &NBSP;ELSE&NBSP;IF&NBSP;EXPRESSION&NBSP;{&NBSP;&NBSP, ...} &NBSP;ELSE&NBSP;{&NBSP;&NBSP, ...} equality, etc: ==, !=, <, >, <=, >=regexp: =~, !~   (regular expression) inclusion: in, not inand, or, nand, xor! #例子如下:filter {   if [action] ==  "Login"  {    mutate { remove =>   "Secret"  }  }}output {  if [type] ==  "Apache"  {     if [status] =~ /^5\d\d/ {      nagios  { ...  }    } else if [status] =~ /^4\d\d/  {      elasticsearch { ... }    }     statsd { increment =>  "Apache.%{status}"  }  }}output {  # Send  production errors to pagerduty  if [loglevel] ==  "ERROR"  and  [deployment] ==  "Production"  {    pagerduty {       ...    }  }}filter {  if [foo] in  [foobar] {    mutate { add_tag =>  "field in  Field " }  }  if [foo] in " foo " {    mutate  { add_tag =>  "field in string"  }  }  if  "Hello " in [greeting] {    mutate { add_tag => " string  in field " }  }  if [foo] in [" Hello ", " World ", " Foo "]&NBSP;{&NBSP;&NBSP;&NBSP;&NBsp;mutate { add_tag =>  "Field in list"  }  }  if  [missing] in [alsomissing] {    mutate { add_tag =>   "Shouldnotexist"  }  }  if ! ("foo"  in ["Hello",  "World"])  {    mutate { add_tag = >  "Shouldexist"  }  }}or, to test if grok was successful:o utput {  if  "_grokparsefailure"  not in [tags] {     elasticsearch { ... }  }}

There are a lot of problems with the alter log before about mutate processing. For example, the original string has more than one: symbol, it will describe the display is not complete. Use Grok to handle the following:

Input{stdin{type = "Hxwtest"}}filter{grok{match = ["Message", "(? <oraerr_id>^o[a-z]{2}-[0-9]{5}):(?" <ora_desc>.*) "]}grok{# (?< Group name >regex) puts the contents of the Regex capture into the group name, and the group name is treated as a field. (? <=:) Surround match + = ["Message", "(?<test> (? <=:). *)"]}if "_grokparsefailure" not in [tags]{mutate{add_  field = {"Ngsubtest" = "%{test}"}} #把TEST中的空格去掉mutate {gsub + = ["TEST", "", ""]}}output{Stdout{codec = Rubydebug}}

The results are as follows:

Ora-01589:alter Database Oracle lkjldkfjdkf{"message" = "Ora-01589:alter database Oracle Lkjldkfjdkf\r", "@version" = "1", "@timestamp" = "2014-12-13t02:50:46.671z", "type" = "Hxwtest", "host"          = "Huangwen", "oraerr_id" and "ORA-01589", "Ora_desc" and "ALTER DATABASE Oracle LKJLDKFJDKF\R", "TEST" = "alterdatabaseoraclelkjldkfjdkf\r", "ngsubtest" = "ALTER DATABASE Oracle LKJLDKFJDKF\R"}

This article from "Despite the wrong, let me wrong to die!" "Blog, be sure to keep this provenance http://hxw168.blog.51cto.com/8718136/1589540

"Logstash"-Logstash Configuration Language Basics