Getting Started with Flume

First, overview:

Flume is a highly available, highly reliable, distributed, massive log collection, aggregation, and transmission system that Cloudera provides.

The current flume has two versions of the Flume 0.9X version of collectively known as the flume-og,flume1.x version of Flume-ng. Since Flume-ng has undergone significant refactoring, it is very different from the flume-og and should be differentiated when used.

Let's take a look at a few streaming diagrams:

As you can see from the above figure, Flume is a tool that imports the source data into the output. The source can have more than one, and the output can have more than one.

Second, the environment description:

The flume used in this article is version 1.6.0.

Three, the Environment construction: (A simple example)

1, first download the corresponding binary tar package from the official website. Http://flume.apache.org/download.html

2. Unzip the file and copy it to the specified directory.

tar -zxvf apache-flume-1.6. 0-bin. tar. gz

3. Create a profile test.conf in the Conf directory under the Flume installation directory:

# example.conf:a single-node Flume configuration# Name The components in this agenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the Sourcea1.sources.r1.type=Netcata1.sources.r1.bind=Localhosta1.sources.r1.port=44444# Describe The Sinka1.sinks.k1.type=logger# use a channelwhichBuffers EventsinchMemorya1.channels.c1.type=memorya1.channels.c1.capacity= +a1.channels.c1.transactionCapacity= -# Bind The source and sink to the Channela1.sources.r1.channels=C1a1.sinks.k1.channel= C1

This configuration file listens on port 44444.

4, start the agent of Flume:

Bin/flume-ng agent--conf conf--conf-file conf/test.conf--name a1-dflume.root.logger=info,console

After executing the above command, the command line will be in the listening state. The data that is heard by the port is printed directly to the command-line window.

The above A1 refers to the A1 defined in configuration file test.conf.

5. Connect via telnet and send the message.

Because of the localhost used in the configuration file, we re-open a command-line window on this machine.

44444

We type the following on the Telnet command line:

At the command line of flume, you can see the data you hear:

At this point, Flume is installed and demonstrates a simple example.

Getting Started with Flume