Flume ng Introduction and Configuration

a common distributed log collection system:
Apache Flume,
Facebook Scribe,
Apache chukwa

1.flume, as a real-time log collection system developed by Cloudera, has been recognized and widely used by the industry. The initial release version of Flume is now collectively known as Flume OG (original Generation), which belongs to Cloudera. But with the expansion of the FLume function, FLume OG code Engineering bloated, the core component design is unreasonable, the core configuration is not standard and other shortcomings exposed, especially in FLume OG final release 0.94.0, log transmission instability is particularly serious, in order to solve these problems, 2011 October 22, Cloudera completed the Flume-728 and made a milestone change to Flume: Refactoring the core components, core configuration, and code architecture, the reconstructed version collectively known as Flume NG (); Another reason for the change is to include Flume in Apache, Cloudera Flume renamed Apache Flume

Features: Flume is a distributed, reliable, and highly available system for collecting, aggregating, and transmitting large volumes of logs. Support for customizing various data senders in the log system for data collection, while Flume provides the ability to simply process the data and write to various data recipients (such as text, HDFS, hbase, etc.)

2,Flume Some of the core concepts:

Components	function
Agent	Run flume using the JVM. Each machine runs an agent, but it can contain multiple sources and sinks in one agent.
Client	Production data, running on a separate thread.
Source	Collect data from the client and pass it to the channel.
Sink	Collects data from the channel and runs on a separate thread.
Channel	Connecting sources and sinks, this is a bit like a queue.
Events	This can be log records, Avro objects, and so on.

Flume is the smallest independent operating unit of the agent. An agent is a JVM. A single agent consists of three components of Source, sink, and channel , such as

Flume ng Introduction and Configuration