Flume Study notes Flume ng overview and single-node installation

Flume ng Overview:

Flume Ng is a distributed, highly available, reliable system that collects, moves, and stores disparate amounts of data into a single data storage system. Lightweight, simple to configure, suitable for a variety of log collections, and supports failover and load balancing. Where the agent contains Source,channel and Sink, three have formed an agent. The duties of the three are as follows:

• Source: Used to consume (collect) the data source into the channel component
• Channel: Interim storage, save all source component information
• Sink: read from channel, delete information in channel after successful reading

Single-node Installation:

• Download tar:
• Extract:

TAR-ZXVF apache-flume-1.7.0-bin.tar.gz

• Set Environment variables:

Export Flume_home=/usr/local/flume

Export path= $PATH: $FLUME _home/bin

• Modify flume-env.sh:

Export java_home=/usr/java/jdk1.7.0_80

• To test whether the installation was successful:

• To modify a configuration file:

a1.sources = R1a1.sinks = K1a1.channels = c1# describe/configure the Sourcea1.sources.r1.type = Execa1.sources.r1.channel s = C1a1.sources.r1.command = Tail-f/home/sky/flume/log_exec_tail# Describe The Sinka1.sinks.k1.type = logger# use a cha Nnel which buffers events in Memorya1.channels.c1.type = Memorya1.channels.c1.capacity = 1000a1.channels.c1.transactioncapacity = 100# Bind The source and sink to the Channela1.sources.r1.channels = C1A1.SINKS.K 1.channel = C1

• Start Agent:

Flume-ng agent-c conf-f/usr/local/flume/conf/exec_tail.conf-n a1-dflume.root.logger=info,console

• Test whether the flume can accept log updates:

Test success!

Flume Study notes Flume ng overview and single-node installation