ELK-Brief talk on Logstash Flume

Now the mainstream log analysis system has Logstash and flume, combined with a lot of online predecessors, summed up a bit, hope and everyone to share and discuss, there are different ideas welcome message.

Flume

Cloudera provides a high-availability, high-reliability, distributed mass log collection, aggregation and transmission system;

Support the customization of various types of data sender, easy to collect data, general and Kafka subscription message system collocation more;

There are currently two versions, OG and ng, the difference is very big, interested can go to study;

Characteristics:

1, focus on data transmission, there is internal mechanism to ensure that data will not be lost for important log scenarios;

2, developed by Java, no rich plug-ins, mainly by two times development;

3, the configuration cumbersome, external exposure monitoring port has data.

650) this.width=650; "src=" Https://s1.51cto.com/wyfs02/M02/8F/B0/wKiom1jp3WvRAZAqAACg0ITYDGM341.png "title=" 1.png "Style=" WIDTH:400PX;HEIGHT:234PX; "alt=" wkiom1jp3wvrazaqaacg0itydgm341.png "vspace=" 0 "hspace=" 0 "height=" 234 " Border= "0" width= "/>"

Logstash

Elastic.co an open source data collection engine that can dynamically unify data from different data sources to destinations;

Objective to process and collect log format, with Elasticsearch for analysis, Kibana for page display;

At present, the latest version 5.3, the integration of the two partners, refer to the official website detailed.

Characteristics:

1, the internal does not have a persist queue, abnormal situation may lose some data;

2, written by Ruby, need Ruby environment, a lot of plugins;

3, emphasis on data pre-processing, easy analysis.

650) this.width=650; "src=" Https://s5.51cto.com/wyfs02/M00/8F/B0/wKiom1jp5cagErcyAAIGKSN7gaA252.png "title=" Qq20170409154059.png "style=" width:400px;height:223px; "alt=" wkiom1jp5cagercyaaigksn7gaa252.png "vspace=" 0 "hspace = "0" height= "223" border= "0" width= "/>"

	Flume	Logstash
Structurally	Source, Channel, Sink	Shipper, Broker, Indexer
Simple degree	Very cumbersome, to separate the source, channel, sink manual configuration, but also involves a complex data acquisition environment	Simple and clear, three parts of the properties are defined, just choose the best, and you can develop the plug-in itself
Historical background	Originally designed to pass data into HDFs, focusing on transport (multi-routing), heavy-stability	Focus on the preprocessing of the data, because the log fields require a lot of preprocessing, to pave the parsing
Contrast	Like the bulk of the desktop, the use of more cumbersome, a wide range of tools, according to Business choice	More like an assembled desktop, easy to use and elk more efficient

This article is from the "North Ice--q" blog, please be sure to keep this source http://beibing.blog.51cto.com/10693373/1914411

ELK-Brief talk on Logstash Flume