Flume ng Overview:
Flume Ng is a distributed, highly available, reliable system that collects, moves, and stores disparate amounts of data into a single data storage system. Lightweight, simple to configure, suitable for a variety of log collections, and supports failover and load balancing. Where the agent contains Source,channel and Sink, three have formed an agent. The duties of the three are as follows:
- Source: Used to consume (collect) the data source into the channel component
- Channel: Interim storage, save all source component information
- Sink: read from channel, delete information in channel after successful reading
Single-node Installation:
TAR-ZXVF apache-flume-1.7.0-bin.tar.gz
- Set Environment variables:
Export Flume_home=/usr/local/flume
Export path= $PATH: $FLUME _home/bin
Export java_home=/usr/java/jdk1.7.0_80
- To test whether the installation was successful:
- To modify a configuration file:
a1.sources = R1a1.sinks = K1a1.channels = c1# describe/configure the Sourcea1.sources.r1.type = Execa1.sources.r1.channel s = C1a1.sources.r1.command = Tail-f/home/sky/flume/log_exec_tail# Describe The Sinka1.sinks.k1.type = logger# use a cha Nnel which buffers events in Memorya1.channels.c1.type = Memorya1.channels.c1.capacity = 1000a1.channels.c1.transactioncapacity = 100# Bind The source and sink to the Channela1.sources.r1.channels = C1A1.SINKS.K 1.channel = C1
Flume-ng agent-c conf-f/usr/local/flume/conf/exec_tail.conf-n a1-dflume.root.logger=info,console
- Test whether the flume can accept log updates:
Test success!
Flume Study notes Flume ng overview and single-node installation