Logstash notes of Distributed log collection (i)

(a) What is Logstash? 

Logstash is a distributed Log collection framework, the development language is JRuby, of course, is to interface with the Java platform, but with Ruby syntax is good, very concise and powerful, often with Elasticsearch,kibana configuration, composed of the famous Elk technology stack, Ideal for analysis of log data.

Of course it can appear alone, as the log collection software, you can collect logs to a variety of storage systems or temporary transit systems, such as MYSQL,REDIS,KAKFA,HDFS, LUCENE,SOLR, etc. is not necessarily elasticsearch.

Official website: Https://www.elastic.co/downloads/logstash 
Official Document Address: https://www.elastic.co/guide/en/logstash/current/index.html 


(ii) installation of the Logstash 

The current version of Logstash is 2.0.0, which is recommended to install on the Linux platform, although it also supports Windows platforms, but there may be problems 
Download: 
wget https://download.elastic.co/logstash/logstash/logstash-2.0.0.tar.gz 

Extract: 
TAR-ZXVF logstash-2.0.0.tar.gz 

Go to the root directory and execute bin/logstash-h to view Help documents 
The parameters are described as follows: 

To use a command template: 
/bin/logstash Command parameter options 

Options: 
-F, specifies that a Logstash configuration module with a suffix of. conf file is loaded 
-E, command line specifying parameters, typically used to debug 
-W, specifying the number of worker threads for Logstash 
-L, specifies that the default log for Logstash is written to a file, and if not specified, the default is standard output 
--quiet silent mode, only error level information output 
--verbose Info-level log output 
--debug the log output of the debug level. 
-V,--version view version of Logstash 
-P,--pluginpath PATH loads the custom Logstash plugin 
-T,--configtest Check if the Logstash configuration is valid 
-H,--help printing help 


(iii) Data processing model of Logstash 

(1) input = "Output" 
(2) input = "Filter =" Output 

Input sources commonly used are: File,syslog,redis,log4j,apache log or Nginx log, or some other custom log format, business log, search log, order Log, etc.

The common options for filter are: 
Grok: Supports regular extraction of any unstructured or structured data, with more than 120 regular logstash built-in, such as common time, IP, username, and so on, to support custom regular parsing 
Mutate: Modify field name, delete, update etc., convert field type, etc. 
Drop: Delete some time, such as debug 
Clone: Copy a copy of the event to add or remove fields 
GeoIP: Get geo-location information via IP, kibana area charts are awesome 
Ruby: Supports native ruby code, operates events, and implements powerful other functions 

Output common outputs are: 
Elasticsearch more commonly used 
File: Writing Files 
Redis: Write queue 
HDFs: Write HDFs, plug-in support required 
Zabbix:zabbix Monitoring 
MongoDB: Writing to the MongoDB library 

Besides, there's a code plugin codecs is also more commonly used 
Often used to process JSON data or multiple rows of data sources 



(iv) Logstash some simple examples 

(1) Debug with command-line commands: 
[Java]  View Plain Copy

2. [Search@h3 logstash-2.0. 0 ]$ bin/logstash-e "input{stdin{}} output{stdout{}}"
4. Default Settings Used:filter workers: 1
6. Logstash Startup completed
8. Hello
10. --04t15:747Z h3 Hello
12. Test
14. --04t15:108Z h3 test

(2) command-line parameters are only suitable for simple configuration, if the configuration is more, we will generally write to a. conf end of the configuration file , and then use
The-f command loads the configuration in (1) and writes the hello.conf

And then use Bin/logstash-f hello.conf to perform the load to achieve the same effect


(3) Common data modules
[Java]  View Plain Copy

2. <pre name="code" class="java">input{
4. .....     
6. }     
8. filter{
10. ......     
12. }     
14. output{
16. .......     
18. }    

(4) Listen file, support wildcard character, write file [Java]  View Plain Copy

2. input{
4.     
6. file = ["/var/log/file","/var/solr/log/*"]
8.     
10. }     
12.     
14. output{
16.     
18. file = "/sys/collect/log"
20.     
22.     
24. }    

(5) Logstash plug-in support data type [Java]  View Plain Copy

2. array: path=> ["A","B"]
4. Boolean: ssl_enable = true
6. bytes:
8. my_bytes ="1113"#1113 bytes
10. my_bytes ="10MiB"#10485760 bytes
12. my_bytes ="100kib"#102400 bytes
14. my_bytes ="MB"#180000000 bytes
16. Code:
18. codec = "JSON"
20. Hash Table:
22. Match + = {
24.     
26. "K1" = " v1 "
28. "K2" = " v2 "
30. "K3" = " v3 "
32.     
34. }     
36. Value:
38. port=>
40. Password:
42. pwd=> "password"
44. Path:
46. path=> "/tmp/logstash"
48. string:
50. name = "Hello wordld"
52. notes:
54. input{
56. # starts with a shell script comment
58. }    

Logstash notes of Distributed log collection (i)