Comparison between Flume and Logstash

Flume compared with Logstash, the personal experience is as follows:

• Logstash more emphasis on the preprocessing of the field, while flume emphasis on data transmission;
• Logstash has dozens of plug-ins, flexible configuration, Flume is to emphasize the user's custom development (source and sink kind also has ten or twenty, the channel is relatively small).
• Buffer is present between the input of the Logstash and the filter and output, and the flume is persisted by using the channel directly (which can be understood as no filter)

First, Logstash briefly:

• Input is responsible for inputting data (generating or collecting, and decoding decode);
• Filter is responsible for the analysis of the collected logs, extract the fields (usually extract the key fields, stored in the Elasticsearch for retrieval analysis);
• Output is responsible for outputting the data to the specified storage location (if it is a collection agent, it is generally sent to the message queue, such as KAFKA,REDIS,MQ; if it is the analysis summary side, it is generally sent to Elasticsearch)

In Logstash, the collaborative work between Input,filter,output is valued, so multiple inputs summarize the data into buffer between input and filter. Filter reads the data from buffer, parses it, and stores it in buffer between the output and the filter. When buffer satisfies a certain condition, the refresh of output is triggered.

Second, Flume Shallowly discusses

In the flume:

• Source is responsible for the same role as input, responsible for data generation or collection (usually docking some RPC programs or other flume node sink)
• Channel is responsible for data storage persistence (usually memory or file two types)
• Sink is responsible for forwarding data (for forwarding to the next flume source or the final storage point-such as HDFs)

Flume more important data transmission, so there is little data parsing preprocessing. Only the data is generated, encapsulated into an event and then transmitted. The transmission of time flume more than Logstash considered some reliability. Because data is persisted in the channel (there are generally two options, Memorychannel is in memory and the other is FileChannel stored in a file), the data is stored only in the next storage location (possibly the final storage location, such as HDFs Or the channel of the next flume node), the data will be removed from the current channel. This process is controlled through a transaction, which guarantees the reliability of the data.

However, the persistence of Flume also has a capacity limit, such as memory if more than a certain amount, also will explode.

See also: http://www.cnblogs.com/xing901022/p/5631445.html

Comparison between Flume and Logstash