Flume NG Introduction and Configuration Combat

Excerpt from: http://my.oschina.net/leejun2005/blog/288136

• 1, Flume Some of the core concepts:
• 1.1 Data flow Model
• 1.2 High reliability
• 1.3 recoverability
• 2, Flume The overall structure of the introduction
• 2.1 Exec Source
• 2.2 Spooling Directory Source
• 3. Common architecture, function configuration Example
• 3.1 First simple: Single-node Flume configuration
• 3.2 Single-node Flume direct write to HDFS
• 3.3来 A common architecture: multi-agent aggregation write HDFS
• 3.3.1 Configuring the Flume Client on each webserv log machine
• 3.3.2 Configuring Flume Server on the aggregation node
• 4. Problems that may be encountered:
• 4.1 OOM Problem:
• 4.2 JDK Version Incompatibility issue:
• 4.3 small files written to HDFS delay problem
• 4.4 Data Repeat write, loss problem
• 4.5 Tail The problem of the continued transmission of breakpoints:
• 4.6 How can I modify, discard, and classify stored data in a predefined rule in Flume?
• 5, Refer:

Copy to Google TranslateTranslation Results2.2Background PrintingDirectory Source

Flume NG Introduction and Configuration Combat