Logstash notes for distributed log Collection (ii) _logstash

Today is November 06, 2015, get up in the morning, Beijing weather unexpectedly snowed, yes, in recent years has rarely seen snow, think of the winter as a child, memories of the shadow is still vivid.

To get to the point, the article introduced the basic knowledge of Logstash and introductory demo, this article introduces several more commonly used commands and cases
Through the previous introduction, we generally know the entire Logstash process log flow:
Input => filter => output

In addition to the above several basic commands, there are ruby,mutate,gork,codes several commands are more commonly used:

Online gork regular Address: http://grokdebug.herokuapp.com/

Logstash Base Regular Address: Https://github.com/elastic/logstash/blob/v1.4.2/patterns/grok-patterns

If you are doing operation, then very lucky, built-in more than 120 regular, for the operational personnel of the very aspects, such as the common Apache log format, nginx log format
The above regular library all has the molding regular formula, has omitted to write the regular one most of the work.

If you are doing development, then a little bit of trouble, development needs to face a variety of business log+ system log, at this time, master Gork usage, the use of regular extracts arbitrary content, become more important, but I believe that most people will be some basis of the regular, so it is relatively easy.

Let's look at a few cases:

Case (a) use Gork to extract some of the field contents from the log of SOLR search:

Example data (note that this is a row of data):

Java code info-2015-11-03 06:10:53.106; [Company] Org.apache.solr.core.SolrCore; [Company] WEBAPP=/SOLR Path=/select params={mm=100%25&sort=score+desc,regcapital+desc,founddate+asc,cpynamepy +asc&start=0&q=+cpyname: Lanzhou top-Tianjin Food Co., Ltd. ++or+sname: "Lanzhou top-Tianjin Food Co., Ltd." ^10+or+oldname: Lanzhou top Tianjin Food Co., Ltd. +&wt=javabin &fq=&version=2&rows=10&deftype=edismax} hits=0 status=0 qtime=2

Using Gork regular syntax + built-in regular, able to extract, log level, UTC time, query keywords and so on content

Java code log level:%{loglevel:loglevel} query time:%{timestamp_iso8601:time}; Query keywords: cpyname: (? <kw>.*) \+\+ hit number: hits= (?
Case (ii) Use the Filter-date plug-in to extract the time inside the log file, overwriting the time that Logstash itself creates the log by default
Website Introduction: https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
This case is also more common, because we need time, is certainly log log inside the time, rather than logstash the time, to ensure the correct time, in order to
Kibana inside the correct display, about +8 UTC time problem, here is not said, search oneself can control to find the correct content can

The following screenshot of the content, wrote a fixed time:





The result is the following figure, which has been converted to UTC time







Case (iii) convert a YYYY-MM-DD HH:mm:ss format date to a long timestamp using ruby inline code

Then the code for case two, and then add the following code inside the filter:




After running, look at the result:



Write a program to verify the correct, found no problem:






If you click JRuby or Ruby syntax, you can use Logstash to do more custom processing tasks

Case (iv) use Codec+multiline to process a log across rows

In what scenario, you need to use the multiline plugin. The most common is that our log4j inside of the log Java program occurs when the exception, often
Throws a whole bunch of exceptions, as follows:






Note that so many lines of log, from the business point of view, it is a row, if the default we do not do any processing, Logstash will parse it into multiple events, so since
Basically deviated from what we expected, then how to deal with it.

Method (1):
In the encoding process of the input phase, add the regular judgment:






Method (2):
or in the input phase, but using the trigger mode, until the next line of logs is encountered, it is simpler to consider all of the contents of the line to the next line, and the advantages may be better in performance and accuracy than the previous one, but the disadvantage is that the next line of logs must be generated, and the current line of log In order to be collected!







: http://blog.sematext.com/2015/05/26/handling-stack-traces-with-logstash/


Case (v) Use Mutate+gsub to remove line breaks in some fields

Java code mutate {gsub => ["message", "\ R", ""]}