Flume Installation and use

Source: Internet
Author: User

Overview

1-flume Introduction

System Requirements

3-Installation and configuration

4-Start and test

I. Introduction to Flume

Website address: http://flume.apache.org/

1-Overview

Flume is a distributed, reliable, and usable service for efficiently collecting, summarizing, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data streams. It has a reliable mechanism of reliability and many failover and recovery mechanisms, with strong fault tolerance and fault tolerance. It uses a simple extensible data model that allows online analysis of applications.

The use of Apache Flume is not limited to log data aggregation. Because data sources are customizable, flume can be used to transfer large amounts of event data, including, but not limited to, network traffic data, social media-generated data, e-mail messages, and almost any possible source of data.

2-Data Model

A Flume event is defined as a unit of data flow has a byte payload and an optional set of string attributes. A Flume agent is a (JVM) process, the components of the through which events flow from a external source to the next D Estination (Hop).-the original sentence of the official website.

As you can see, 1 agents need to have three parts: source, channel, sink, system requirements
    1. Java Runtime Environment-Java version 1.8 or later
    2. Memory-source, channel or receiver uses a configuration that has sufficient memory
    3. Disk space-sufficient disk space for the configuration used by the channel or receiver
    4. Directory permissions-read/write permissions for the directory used by the agent
Three, installation and configuration: http://flume.apache.org/download.html Select the latest version 1.8
Upload to the/lcoal directory and view the file size (56M)
[Email protected] local]# CD/du -sm apache-flume-1.8. 0-bin. Tar . GZ  apache-flume-1.8. 0-bin. tar. gz

Unzip and move to the/OPT

MV apache-flume-1.8. 0-mv flume//opt/

Use one of the official website configuration:
[Email protected] flume]# CD conf/cp flume-conf.properties.template flume-conf.properties[[ Email protected] conf]# vim flume-conf.properties

The configuration content is as follows:

# example.conf:a single-node Flume configuration# Name The components in this agenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the Sourcea1.sources.r1.type=Netcata1.sources.r1.bind=Localhosta1.sources.r1.port=44444# Describe The Sinka1.sinks.k1.type=logger# use a channelwhichBuffers EventsinchMemorya1.channels.c1.type=memorya1.channels.c1.capacity= +a1.channels.c1.transactionCapacity= -# Bind The source and sink to the Channela1.sources.r1.channels=C1a1.sinks.k1.channel= C1

This is set to memory, read the RAM, through the Telnet localhost 44444 port connection under the session information will be flume monitored.

Test

Four-start and test

Start Flume:

[[email protected] flume]# bin/flume-ng agent--conf conf--conf-file conf/flume-conf.properties--name A1-DFLUME.ROOT.L Ogger=info,console

Create another session

[[Email protected] ~]# telnet localhost 44444
-bash:telnet:command not found

[email protected] ~]# Yum Install

......

Installed:
Telnet.x86_64 1:0.17-48.el6

complete!

To view Flume connected sessions

This information is captured.

Indicates that the flume is functioning properly.

The above is just a simple test.

More plug-ins, multi-agent, and so on, and then continue.

Flume Installation and use

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.