Flume Introduction Sample

Flume, as a real-time log collection system developed by Cloudera, has been recognized and widely used by the industry. The initial release version of Flume is now collectively known as Flume OG (original Generation), which belongs to Cloudera. But with the expansion of the FLume function, FLume OG code Engineering bloated, the core component design is unreasonable, the core configuration is not standard and other shortcomings exposed, especially in FLume OG final release 0.94.0, log transmission instability is particularly serious, in order to solve these problems, 2011 October 22, Cloudera completed the Flume-728 and made a milestone change to Flume: Refactoring the core components, core configuration, and code architecture, the reconstructed version is collectively known as Flume NG (Next generation), and another reason for the change is Flume Included in Apache, Cloudera Flume renamed Apache Flume.

Http://flume.apache.org/index.html

(The above image is from the official website document)

Some core concepts of flume:

The Agent uses the JVM to run Flume. Each machine runs an agent, but it can contain multiple sources and sinks in one agent.
The Client produces data that runs on a separate thread.
Source collects data from the client and passes it to the channel.
Sink collects data from the channel and runs on a separate thread.
The Channel connects sources and sinks, which is a bit like a queue.
Events can be log records, Avro objects, and so on.

The flume has a high level of scalability that can be arbitrarily combined:
Note that source is the sink that is the sending source

Download-Unzip-Modify/etc/profile

Example: File append content as message input

EXEC executes a given command to get the source of the output, and if you want to use the tail command, the required file is large enough to see the output

1. Create Profile VI example.conf

A1.sources =r1a1.sinks=K1a1.channels=C1#describe/configure the sourceA1.sources.r1.type =execA1.sources.r1.channels=C1a1.sources.r1.command= Tail-f ~/test/Log_exec_tail#Describe the sinkA1.sinks.k1.type =Logger#Use a channel which buffers events in memoryA1.channels.c1.type =memorya1.channels.c1.capacity= 1000a1.channels.c1.transactionCapacity= 100#Bind the source and sink to the channelA1.sources.r1.channels =C1a1.sinks.k1.channel= C1

2. Start the flume agent

Flume-ng agent-c conf-  f ~/test/example.conf--name A1-dflume.root.logger=info,console

3. Append the file

 for inch {1..  - };      Do Echo " exec tail$i " >> ~/test/log_exec_tail;  Done

4. At the start of the Flume Agent console, you can see that the additional information appears

More as into the type See also: http://www.aboutyun.com/thread-8917-1-1.html

Flume Introduction Sample