Teach you one step to build flume distributed log system

How do I collect processing in the previous dozens of lines of Business Journal system? has introduced the flume of the numerous application scenarios, then this article first describes how to build a single version of the log system.

Environment

CentOS7.0

Java1.8

Download

Official website Download http://flume.apache.org/download.html

Current Latest Version apache-flume-1.7.0-bin.tar.gz

Download and upload to the/usr/local/folder in CentOS and unzip to the current file to rename to flume170/usr/local/flume170

TAR-ZXVF apache-flume-1.7.0-bin.tar.gz

Installation configuration

Modify flume-env.sh configuration file, mainly add Java_home variable settings

Java_home=/usr/lib/jvm/java8

Set global variables for flume

Open profile

Vi/etc/profile

Add to

Export flume=/usr/local/flume170export PATH= $PATH: $FLUME/bin

Then make the environment variable effective

Source/etc/profile

Verify that the installation is successful

Flume-ng version

Test Small instances

Example of a reference spool type on the Web

Spool monitors the configuration of new files in the directory and reads the data from the files. There are two points to note:
1) files copied to the spool directory can no longer be opened for editing.
2) Spool directory cannot contain the corresponding subdirectory
To create an agent configuration file

#vi/usr/local/flume170/conf/spool.confA1.sources =R1a1.channels=c1a1.sinks=K1#describe/configure the sourceA1.sources.r1.type =Spooldira1.sources.r1.channels=C1a1.sources.r1.spoolDir=/usr/local/flume170/Logsa1.sources.r1.fileHeader=true#Use a channel which buffers events in memoryA1.channels.c1.type =memorya1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100#Describe the sinkA1.sinks.k1.type =Loggera1.sinks.k1.channel= C1

Spooldir: Set the Monitoring folder, when there is a file entry will read the contents of the file and sent through sink, after the completion of the message after the file name is added. Complete

Start Flume Agent A1

/usr/local/flume170/bin/flume-ng agent-c. -f/usr/local/flume170/conf/spool.conf-n A1-dflume.root.logger=info,console

Append a file to the/usr/local/flume170/logs directory

# echo "Spool test1" >/usr/local/flume170/logs/spool_text.log

In the console, you can see the following related information:

14/08/10 11:37:13INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown.14/08/10 11:37:13INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown.14/08/10 11:37:14 INFO Avro. Reliablespoolingfileeventreader:preparing to move File/usr/local/flume170/logs/spool_text.log to/usr/local/ flume170/logs/Spool_text.log.COMPLETED14/08/10 11:37:14INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown. 14/08/10 11:37:14INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown. 14/08/10 11:37:14 INFO sink. Loggersink:event:{headers:{file=/usr/local/flume170/logs/spool_text.log} body:73 6F 6F 6C spool test1} 14/08/10 11:37:15INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown. 14/08/10 11:37:15INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown. 14/08/10 11:37:16INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown. 14/08/10 11:37:16INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown. 14/08/10 11:37:17 INFO Source. Spooldirectorysource:spooling Directory Source Runner has shutdown.

What appears above indicates that it is ready to run, the entire installation process is simple, mainly configuration.

As for the distributed need to set source and sink.

For example, the log generated by flume in each business is then used by a flume to receive the rollup, and then the aggregated log is sent uniformly to Kafka for unified processing and finally saved to HDFs or hbase. , the flume in each business can be load and master, which can be seen as highly scalable.

Teach you one step to build flume distributed log system