Kafka and Flume

Https://www.ibm.com/developerworks/cn/opensource/os-cn-kafka/index.html

Kafka and Flume Many of the functions are really repetitive. Here are some suggestions for evaluating the two systems:

1. Kafka is a general-purpose system. You can have many producers and consumers to share multiple themes. Conversely, Flume is designed to work for a specific purpose and is sent specifically to HDFS and HBase. Flume is optimized to better serve HDFS and is integrated with the security architecture of Hadoop. Based on this conclusion, the Hadoop developer Cloudera recommends using Kafka if the data needs to be consumed by multiple applications, and Flume can be used if the data is only for Hadoop.

2. The Flume has many configuration sources (sources) and storage pools (sinks). Then, Kafka has a very small producer and consumer environment system, and the Kafka community is not very supportive of this. If your data sources have been identified and do not require additional coding, then you can use the sources and sinks provided by Flume, and conversely, if you need to prepare your own producers and consumers, then you need to use Kafka.

3. Flume can process data in real time in interceptors. This feature is useful for filtering data. Kafka requires an external system to help process the data.

4. Both systems can guarantee no loss of data, either Kafka or Flume. Then, Flume does not replicate the event. Accordingly, even if we are using a trusted file channel, if the node where the Flume agent is located goes down, you will lose all of the event access until you fix the damaged node. There is no such problem with pipe features that use Kafka.

5. Flume and Kafka can work together. If you need to transfer streaming data from Kafka to Hadoop, you can use the Flume Agent (agent) to treat Kafka as a source (source), which can read data from Kafka to Hadoop. You don't need to develop your own consumers, you can use Flume with Hadoop, HBase, use the Cloudera Manager platform to monitor consumers, and process data by adding filters.

Kafka and Flume