Log Collection System Flume research note 2nd-flume configuration and usage examples

The previous note describes the usage scenarios and system architecture of Flume, and this note takes an example of how flume is configured. The text begins below.

1. Flume usage Examples
1.1 Configuration
The 3 components of the Flume agent and their topological relationships are specified in the configuration file, and the general principle is that important configuration items such as Source/channel/sink's Name/type must be listed, and the source (s) and sink (s) are connected through the channel. , in addition,1 source can specify multiple channel, while 1 sink can only receive data from 1 channel .
Here is a sample configuration for deploying 1 sets of flume agents with 1 source, 1 channel, 1 sink, and more configuration rules to refer to the Configurations section of Flume User Guide.
Assuming that the external data source is Nginx Access_log, to ensure that the data is not lost (it is simpler to perform tail–f/path/to/access_log through Execsource, but the data is lost when the flume-agent process is abnormal), We choose spooling Directory Source Type source, choose the memory type of channel, choose the File_roll type of sink for debugging.
In addition, we use the Logrotate tool to periodically slice the nginx access_log and move the segmented file into the spooling directory source directory so that the flume source detects and processes the file.
The configuration file that meets the above requirements is shown below.

agent_test.sources = Spool-srcagent_test.channels = Mem-chanagent_test.sinks = Logger-sinkagent_ Test.sources.spool-src.type = Spooldiragent_test.sources.spool-src.channels = Mem-chanagent_ Test.sources.spool-src.spooldir =/home/slvher/tools/nginx-logs.bakagent_test.sinks.logger-sink.type = File_ Rollagent_test.sinks.logger-sink.channel = Mem-chanagent_test.sinks.logger-sink.sink.directory =/home/slvher/ Tools/flume-1.5.2/apache-flume-1.5.2-bin/flume_sink.dumpagent_test.sinks.logger-sink.sink.rollinterval = 3600agent_test.channels.mem-chan.type = Memoryagent_test.channels.mem-chan.capacity = 10000

Note: The use of the Logrotate tool has been mentioned in a note before, which is skipped here.

1.2 Test
After the configuration is complete, start flume, and refer to the following command:

Export classpath=/home/slvher/tools/flume-1.5.2/apache-flume-1.5.2-bin/lib/*.jar:/home/slvher/tools/ Hadoop-client/java6/lib/*.jarnohup./bin/flume-ng agent-n agent_test-c conf-f conf/flume-conf.properties-f conf/ Log4j.properties > Logs/start.log 2>&1 &

Note:If you start flume by using the./bin/flume-ng script, you encounter the following Shell syntax error

Bin/flume-ng:line 102:syntax Error in conditional expression:unexpected token ' (' Bin/flume-ng:line 102:syntax error n Ear ' slf4j-(a ' bin/flume-ng:line 102: ' If [[$PIECE =~ slf4j-(API|LOG4J12)] [*\.jar]]; Then '

You need to modify the IF condition expression at the prompt line ( enclose regular expressions in double quotation marks), will be the original
if [[$PIECE =~ slf4j-(API|LOG4J12). *\.jar]]; Then
Revision changed to
if [[$PIECE =~"slf4j-(API|LOG4J12). *\.jar"]]; Then
After the boot is complete, you can view the Logs/flume.log log and, if there is no warn or error, it indicates that the flume started successfully.
We will logrotate to the Nginx access_log log segmentation action set to cycle 5minutes of the scheduled task, every 5 minutes, The Nginx-logs.bak directory generates 1 shard files, which are scanned and processed by the source of the Flume and appended to the original file when processing is complete. Completed "suffix.
Since the type we configured for sink is File_roll, after the log is passed through Source-channel-sink, it dumps to the Flume_sink.dump directory where it can verify that the dump was successful.
In real business, the type of sink can be configured as a sink for HDFS or other storage systems as required, and sink writes event data to the corresponding storage system, which is not described here.

2. Monitoring
According to the Flume documentation about monitoring, its monitoring mechanism is still being perfected, the mechanism currently available is to start the flume agent through-D to specify the monitoring of the Server:port, details to see the document.

Resources
1. Flume User guide-configuration
2. Flume User guide-monitoring

========================= EOF =========================

Log Collection System Flume research note 2nd-flume configuration and usage examples