Logstash Multiline plugin, matching multiple lines of log

In addition to accessing the log, the log is processed, which is written mostly by programs, such as log4j. The most important difference between a run-time log and an access log is that the runtime logs are multiple lines, that is, multiple lines in a row can express a meaning.

In filter, add the following code:

Filter {

Multiline {}

}

If you can do it on multiple lines, it is easy to split them into fields.

Field Properties:

For multiline plug-ins, there are three settings that are important: negate, pattern and what

Negate: Type is Boolean default is False

Pattern

Must be set, and no default value, type string, to match the next expression

WHAT:

Must be set, and there is no default value, can be previous (previous) or Next

Here's a look at this example:

# Cat logstash_multiline_shipper.conf Input {file {path="/apps/logstash/conf/test/c.out"type="Runtimelog"codec=Multiline {pattern="^\["negate=true What="previous"} start_position="beginning"Sincedb_path="/apps/logstash/logs/sincedb-access"Ignore_older=0}}output {stdout{codec=Rubydebug}}

Description: The area with "[" line, if not, it must belong to the previous line

The test data is as follows:

[ --Geneva- A Geneva: +: onDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +: GenevaDEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS asc[ --Geneva- A Geneva: +:GenevaDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +:GenevaDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +: toDEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS desc[ --Geneva- A Geneva: +: .DEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS asc[ --Geneva- A Geneva: +: -DEBUG] model. mappingnode:-['/store/shopclass'] matched over.

Start Logstash:

# ./.. /bin/logstash-F logstash_multiline_shipper.conf sending Logstash's logs to/apps/logstash/logs which is now configured via Log4j2.properties[ .- A-09T15: -: -,173][info][logstash.pipeline] Starting pipeline {"ID"="Main","pipeline.workers"=4,"pipeline.batch.size"= the,"Pipeline.batch.delay"=5,"Pipeline.max_inflight"= -}[ .- A-09T15: -: -,192][info][logstash.pipeline] Pipeline main started[ .- A-09T15: -: -,263][info][logstash.agent] Successfully started Logstash API endpoint {:p ort=>9601}

After adding the test data to the log being monitored, check the output:

# ./.. /bin/logstash-F logstash_multiline_shipper.conf sending Logstash's logs to/apps/logstash/logs which is now configured via Log4j2.properties[ .- A-09T15: -: -,173][info][logstash.pipeline] Starting pipeline {"ID"="Main","pipeline.workers"=4,"pipeline.batch.size"= the,"Pipeline.batch.delay"=5,"Pipeline.max_inflight"= -}[ .- A-09T15: -: -,192][info][logstash.pipeline] Pipeline main started[ .- A-09T15: -: -,263][info][logstash.agent] Successfully started Logstash API endpoint {:p ort=>9601}{          "Path"="/apps/logstash/conf/test/c.out",    "@timestamp"= .- A-09t07: +: the. 403Z,"@version"="1",          "Host"="OFS1",       "message"="# ./..  /bin/logstash-f logstash_multiline_shipper.conf \nsending logstash ' s logs to/apps/logstash/logs which are now configured Via Log4j2.properties",          "type"="Runtimelog",          "Tags"= [        [0]"Multiline"    ]}{          "Path"="/apps/logstash/conf/test/c.out",    "@timestamp"= .- A-09t07: +: the. 409Z,"@version"="1",          "Host"="OFS1",       "message"="[2016-12-09t15:16:59,173][info][logstash.pipeline] starting pipeline {\ "id\" =>\ "main\", \ "Pipeline.workers \ "=>4, \" pipeline.batch.size\ "=>125, \" pipeline.batch.delay\ "=>5, \" Pipeline.max_inflight\ "=>500} ",          "type"="Runtimelog",          "Tags"= []}{          "Path"="/apps/logstash/conf/test/c.out",    "@timestamp"= .- A-09t07: +: the. 410Z,"@version"="1",          "Host"="OFS1",       "message"="[2016-12-09t15:16:59,192][info][logstash.pipeline] Pipeline main started",          "type"="Runtimelog",          "Tags"= []}

Logstash Multiline plugin, matching multiple lines of log