Getting Started with Flume

Source: Internet
Author: User

First, overview:

Flume is a highly available, highly reliable, distributed, massive log collection, aggregation, and transmission system that Cloudera provides.

The current flume has two versions of the Flume 0.9X version of collectively known as the flume-og,flume1.x version of Flume-ng. Since Flume-ng has undergone significant refactoring, it is very different from the flume-og and should be differentiated when used.

Let's take a look at a few streaming diagrams:

As you can see from the above figure, Flume is a tool that imports the source data into the output. The source can have more than one, and the output can have more than one.

Second, the environment description:

The flume used in this article is version 1.6.0.

Three, the Environment construction: (A simple example)

1, first download the corresponding binary tar package from the official website. Http://

2. Unzip the file and copy it to the specified directory.

tar -zxvf apache-flume-1.6. 0-bin. tar. gz

3. Create a profile test.conf in the Conf directory under the Flume installation directory:

# example.conf:a single-node Flume configuration# Name The components in this agenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the Sourcea1.sources.r1.type=Netcata1.sources.r1.bind=Localhosta1.sources.r1.port=44444# Describe The Sinka1.sinks.k1.type=logger# use a channelwhichBuffers EventsinchMemorya1.channels.c1.type=memorya1.channels.c1.capacity= +a1.channels.c1.transactionCapacity= -# Bind The source and sink to the C1

This configuration file listens on port 44444.

4, start the agent of Flume:

Bin/flume-ng agent--conf conf--conf-file conf/test.conf--name a1-dflume.root.logger=info,console

After executing the above command, the command line will be in the listening state. The data that is heard by the port is printed directly to the command-line window.

The above A1 refers to the A1 defined in configuration file test.conf.

5. Connect via telnet and send the message.

Because of the localhost used in the configuration file, we re-open a command-line window on this machine.


We type the following on the Telnet command line:

At the command line of flume, you can see the data you hear:

At this point, Flume is installed and demonstrates a simple example.

Getting Started with Flume

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.