Logstash Multiline plugin, matching multiple lines of log

Source: Internet
Author: User
Tags logstash

In addition to accessing the log, the log is processed, which is written mostly by programs, such as log4j. The most important difference between a run-time log and an access log is that the runtime logs are multiple lines, that is, multiple lines in a row can express a meaning.

In filter, add the following code:

Filter {

Multiline {}

}

If you can do it on multiple lines, it is easy to split them into fields.

Field Properties:

For multiline plug-ins, there are three settings that are important: negate, pattern and what

Negate: Type is Boolean default is False

Pattern

Must be set, and no default value, type string, to match the next expression

WHAT:

Must be set, and there is no default value, can be previous (previous) or Next

Here's a look at this example:

# Cat logstash_multiline_shipper.conf Input {file {path="/apps/logstash/conf/test/c.out"type="Runtimelog"codec=Multiline {pattern="^\["negate=true What="previous"} start_position="beginning"Sincedb_path="/apps/logstash/logs/sincedb-access"Ignore_older=0}}output {stdout{codec=Rubydebug}}

Description: The area with "[" line, if not, it must belong to the previous line

The test data is as follows:

[ --Geneva- A Geneva: +: onDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +: GenevaDEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS asc[ --Geneva- A Geneva: +:GenevaDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +:GenevaDEBUG] model. mappingnode:-['/store/shopclass'] matched over. [ --Geneva- A Geneva: +: toDEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS desc[ --Geneva- A Geneva: +: .DEBUG] Impl. jdbcentityinserter:- fromproduct_category product_categorywhereProduct_category. parent_id is NULLand product_category. STATUS =? and product_category. dealer_id is NULLORDER by product_category. ORDERS asc[ --Geneva- A Geneva: +: -DEBUG] model. mappingnode:-['/store/shopclass'] matched over.

Start Logstash:

# ./.. /bin/logstash-F logstash_multiline_shipper.conf sending Logstash's logs to/apps/logstash/logs which is now configured via Log4j2.properties[ .- A-09T15: -: -,173][info][logstash.pipeline] Starting pipeline {"ID"="Main","pipeline.workers"=4,"pipeline.batch.size"= the,"Pipeline.batch.delay"=5,"Pipeline.max_inflight"= -}[ .- A-09T15: -: -,192][info][logstash.pipeline] Pipeline main started[ .- A-09T15: -: -,263][info][logstash.agent] Successfully started Logstash API endpoint {:p ort=>9601}

After adding the test data to the log being monitored, check the output:

# ./.. /bin/logstash-F logstash_multiline_shipper.conf sending Logstash's logs to/apps/logstash/logs which is now configured via Log4j2.properties[ .- A-09T15: -: -,173][info][logstash.pipeline] Starting pipeline {"ID"="Main","pipeline.workers"=4,"pipeline.batch.size"= the,"Pipeline.batch.delay"=5,"Pipeline.max_inflight"= -}[ .- A-09T15: -: -,192][info][logstash.pipeline] Pipeline main started[ .- A-09T15: -: -,263][info][logstash.agent] Successfully started Logstash API endpoint {:p ort=>9601}{          "Path"="/apps/logstash/conf/test/c.out",    "@timestamp"= .- A-09t07: +: the. 403Z,"@version"="1",          "Host"="OFS1",       "message"="# ./..  /bin/logstash-f logstash_multiline_shipper.conf \nsending logstash ' s logs to/apps/logstash/logs which are now configured Via Log4j2.properties",          "type"="Runtimelog",          "Tags"= [        [0]"Multiline"    ]}{          "Path"="/apps/logstash/conf/test/c.out",    "@timestamp"= .- A-09t07: +: the. 409Z,"@version"="1",          "Host"="OFS1",       "message"="[2016-12-09t15:16:59,173][info][logstash.pipeline] starting pipeline {\ "id\" =>\ "main\", \ "Pipeline.workers \ "=>4, \" pipeline.batch.size\ "=>125, \" pipeline.batch.delay\ "=>5, \" Pipeline.max_inflight\ "=>500} ",          "type"="Runtimelog",          "Tags"= []}{          "Path"="/apps/logstash/conf/test/c.out",    "@timestamp"= .- A-09t07: +: the. 410Z,"@version"="1",          "Host"="OFS1",       "message"="[2016-12-09t15:16:59,192][info][logstash.pipeline] Pipeline main started",          "type"="Runtimelog",          "Tags"= []}

Logstash Multiline plugin, matching multiple lines of log

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.