In addition to accessing the log, the log is processed, which is written mostly by programs, such as log4j. The most important difference between a run-time log and an access log is that the runtime logs are multiple lines, that is, multiple lines in a row can express a meaning.
In filter, add the following code:
Filter {
Multiline {}
}
If you can do it on multiple lines, it is easy to split them into fields.
Field Properties:
For multiline plug-ins, there are three settings that are important: negate, pattern and what
Negate: Type is Boolean default is False
Pattern
Must be set, and no default value, type string, to match the next expression
WHAT:
Must be set, and there is no default value, can be previous (previous) or Next
Here's a look at this example:
# Cat logstash_multiline_shipper.conf Input {file {path="/apps/logstash/conf/test/c.out"type="Runtimelog"codec=Multiline {pattern="^\["negate=true What="previous"} start_position="beginning"Sincedb_path="/apps/logstash/logs/sincedb-access"Ignore_older=0}}output {stdout{codec=Rubydebug}}
Description: The area with "[" line, if not, it must belong to the previous line
The test data is as follows:
[ --Geneva- A Geneva: +: onDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +: GenevaDEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS asc[ --Geneva- A Geneva: +:GenevaDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +:GenevaDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +: toDEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS desc[ --Geneva- A Geneva: +: .DEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS asc[ --Geneva- A Geneva: +: -DEBUG] model. mappingnode:-['/store/shopclass'] matched over.
Start Logstash:
# ./.. /bin/logstash-F logstash_multiline_shipper.conf sending Logstash's logs to/apps/logstash/logs which is now configured via Log4j2.properties[ .- A-09T15: -: -,173][info][logstash.pipeline] Starting pipeline {"ID"="Main","pipeline.workers"=4,"pipeline.batch.size"= the,"Pipeline.batch.delay"=5,"Pipeline.max_inflight"= -}[ .- A-09T15: -: -,192][info][logstash.pipeline] Pipeline main started[ .- A-09T15: -: -,263][info][logstash.agent] Successfully started Logstash API endpoint {:p ort=>9601}
After adding the test data to the log being monitored, check the output:
# ./.. /bin/logstash-F logstash_multiline_shipper.conf sending Logstash's logs to/apps/logstash/logs which is now configured via Log4j2.properties[ .- A-09T15: -: -,173][info][logstash.pipeline] Starting pipeline {"ID"="Main","pipeline.workers"=4,"pipeline.batch.size"= the,"Pipeline.batch.delay"=5,"Pipeline.max_inflight"= -}[ .- A-09T15: -: -,192][info][logstash.pipeline] Pipeline main started[ .- A-09T15: -: -,263][info][logstash.agent] Successfully started Logstash API endpoint {:p ort=>9601}{ "Path"="/apps/logstash/conf/test/c.out", "@timestamp"= .- A-09t07: +: the. 403Z,"@version"="1", "Host"="OFS1", "message"="# ./.. /bin/logstash-f logstash_multiline_shipper.conf \nsending logstash ' s logs to/apps/logstash/logs which are now configured Via Log4j2.properties", "type"="Runtimelog", "Tags"= [ [0]"Multiline" ]}{ "Path"="/apps/logstash/conf/test/c.out", "@timestamp"= .- A-09t07: +: the. 409Z,"@version"="1", "Host"="OFS1", "message"="[2016-12-09t15:16:59,173][info][logstash.pipeline] starting pipeline {\ "id\" =>\ "main\", \ "Pipeline.workers \ "=>4, \" pipeline.batch.size\ "=>125, \" pipeline.batch.delay\ "=>5, \" Pipeline.max_inflight\ "=>500} ", "type"="Runtimelog", "Tags"= []}{ "Path"="/apps/logstash/conf/test/c.out", "@timestamp"= .- A-09t07: +: the. 410Z,"@version"="1", "Host"="OFS1", "message"="[2016-12-09t15:16:59,192][info][logstash.pipeline] Pipeline main started", "type"="Runtimelog", "Tags"= []}
Logstash Multiline plugin, matching multiple lines of log