Log collection and processing framework------[Logstash] Use detailed

The Logstash is a lightweight Log collection processing framework that allows you to easily collect scattered, diverse logs and customize them for processing, and then transferring them to a specific location, such as a server or file.

This article is for the Official document translation and practice, I hope there are more users to understand, use this tool. download, install, use

This tool is out-of-the-box software, download address stamp here, download their corresponding system version can be.

After downloading the direct decompression, you can.

From the command line, go to the Logstash/bin directory and execute the following command:

Logstash-e ""

You can see the following information (This command is described later), enter Hello world!

Can see Logstash tail we automatically added several fields, timestamp @timestamp, version @version, type of input, and hostname host. Working principle

Logstash uses the pipeline method to collect and process logs. A bit like *nix system pipe command XXX | ccc | DDD, XXX executes the CCC and then executes DDD.

In Logstash, there are three stages:

input---process filter (not required)--outputs output

Each phase is worked with a number of plugins, such as file, Elasticsearch, Redis, and so on.

Each stage can also be specified in a variety of ways, such as output can be output to elasticsearch, or can be specified to stdout in the console printing.

Thanks to this plug-in organization, Logstash becomes easy to scale and customize. Commands that are commonly used on the command line

-F: This command allows you to specify the configuration file for the Logstash, based on the configuration file configuration Logstash

-E: followed by the string, which can be used as the configuration of the Logstash (if "" The default is to use stdin as input, stdout as output)

-L: The address of the log output (the default is stdout directly in the console output)

-T: Test the configuration file correctly, and then exit.

configuration File Description

As mentioned earlier, Logstash basically consists of three parts, input, output, and filter that the user needs to add, so the standard configuration file format is as follows:

Input {...} 
Filter {...} 
Output {...}

　　

In each section, you can also specify multiple access methods, for example, if I want to specify two log source files, you can write:

Input { 
 file {path = "/var/log/messages" type = "syslog"} 
 file {path = "/var/log/apache/access.log" Type = "Apache"} 
}

Similarly, if more than one processing rule is added to the filter, it is processed in order one by one, but some plugins are not thread-safe.

For example, you specify two plug-ins in the filter, which are not guaranteed to be executed exactly in order, so it is also recommended to avoid reusing plugins in the filter.

To finish this, simply create a small example of a configuration file to see:

Input {
    file {
#指定监听的文件路径, note must be an absolute path, path
        ' = ' e:/software/logstash-1.5.4/logstash-1.5.4/data/ Test.log "
        start_position = Beginning
    }
}
filter {
    
}
output {
    stdout {}
}

The log is roughly as follows:

1 Hello,this is first line in test.log!
2 hello,my name is Xingoo!
3 Goodbye.this is the last line in test.log!

Note that there is a blank line at the end.

The execution command gets the following information:

Careful will find that this log output is not the same as the above Logstash-e "", this is because the above command by default specifies that the returned format is in JSON form.

At this point, is the introduction of Logstash introductory article, will be introduced later about the Logstash more content, interested can pay attention to OH. Reference

"1" logstash Official document: https://www.elastic.co/guide/en/logstash/current/index.html