Use of the Logstash filter

Recently in the project using Logstash do log collection and filtering, feel logstash is still very powerful.

Input {file{path = "/xxx/syslog.txt" Start_position = beginning codec = Multilin e{Patterns_dir = ["/xx/logstash-1.5.3/patterns"] pattern = "^%{message}" Nega        Te = True what = "previous"}}}filter{mutate{split = ["message", "|"] Add_field = {"tmp" = "%{[message][0]}"} Add_field + = {"Deviceproduc         T "= ="%{[message][2]} "} Add_field = {" Deviceversion "="%{[message][3]} "}            Add_field = {"Signature ID" = "%{[message][4]}"} Add_field + = {            "Name" = "%{[message][5]}"} mutate{split + = ["tmp", ":"] Add_field + =        "Tmp1" = "%{[tmp][1]}"} Add_field = {"Version" = "%{[tmp][2]}"} Remove_field = ["TMP "]} grok{patterns_dir = ["/xxx/logstash-1.5.3/patterns "] match + = {" Tmp1 "="%{type:type} " "} Remove_field = [" Tmp1 "]} kv{Include_keys = [" EventId "," msg "," End "," MRT "," Modelconfiden Ce "," severity "," Relevance "," assetcriticality "," priority "," art "," RT "," CS1 "," CS2 "," CS3 "," locality "," Cs2label "," Cs3label "," Cs4label "," Flexstring1label "," Ahost "," AGT "," AV "," Atz "," Aid "," at "," DVC "," Devicezoneid "," Devicezoneuri "," DTZ "," Eventannotationstageupdatetime "," Eventannotationmodificationtime "," Eventannotationaudittrail "," Eventannotationversion "," Eventannotationflags "," Eventannotationendtime "," Eventannotationmanagerreceipttime "," _cefver "," Ad.arcsighteventpath "]} mutate{split = [" Ad.arcsighteventpat H ",", "] Add_field = {" Arcsighteventpath "="%{[ad.arcsighteventpath][0]} "} REM        Ove_field = ["Ad.arcsighteventpath"] Remove_field = ["message"]}}output{kafka{topic_id = "Rawlog" batch_num_messages = Broker_list = "10.3.162.193:39192,10.3.162.194:39192 , 10.3.162.195:39192 "codec =" JSON "} stdout{codec = Rubydebug}

Input: Access data source

Filter : filtering the data source

Output : the

One of the most important is the processing of the filter , and our requirement now is to extract the string key-value.

1, the use of split in the mutate, can be divided by the division of the processing.

2. Use regular grok to intercept the string.

3. Use KV to extract all Key-value

Use of the Logstash filter