Flume Installation and use

Overview

1-flume Introduction

System Requirements

3-Installation and configuration

4-Start and test

I. Introduction to Flume

Website address: http://flume.apache.org/

1-Overview

Flume is a distributed, reliable, and usable service for efficiently collecting, summarizing, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data streams. It has a reliable mechanism of reliability and many failover and recovery mechanisms, with strong fault tolerance and fault tolerance. It uses a simple extensible data model that allows online analysis of applications.

The use of Apache Flume is not limited to log data aggregation. Because data sources are customizable, flume can be used to transfer large amounts of event data, including, but not limited to, network traffic data, social media-generated data, e-mail messages, and almost any possible source of data.

2-Data Model

A Flume event is defined as a unit of data flow has a byte payload and an optional set of string attributes. A Flume agent is a (JVM) process, the components of the through which events flow from a external source to the next D Estination (Hop).-the original sentence of the official website.

As you can see, 1 agents need to have three parts: source, channel, sink, system requirements

1. Java Runtime Environment-Java version 1.8 or later
2. Memory-source, channel or receiver uses a configuration that has sufficient memory
3. Disk space-sufficient disk space for the configuration used by the channel or receiver
4. Directory permissions-read/write permissions for the directory used by the agent

Three, installation and configuration: http://flume.apache.org/download.html Select the latest version 1.8
Upload to the/lcoal directory and view the file size (56M)

[Email protected] local]# CD/du -sm apache-flume-1.8. 0-bin. Tar . GZ  apache-flume-1.8. 0-bin. tar. gz

Unzip and move to the/OPT

MV apache-flume-1.8. 0-mv flume//opt/

Use one of the official website configuration:

[Email protected] flume]# CD conf/cp flume-conf.properties.template flume-conf.properties[[ Email protected] conf]# vim flume-conf.properties

The configuration content is as follows:

# example.conf:a single-node Flume configuration# Name The components in this agenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the Sourcea1.sources.r1.type=Netcata1.sources.r1.bind=Localhosta1.sources.r1.port=44444# Describe The Sinka1.sinks.k1.type=logger# use a channelwhichBuffers EventsinchMemorya1.channels.c1.type=memorya1.channels.c1.capacity= +a1.channels.c1.transactionCapacity= -# Bind The source and sink to the Channela1.sources.r1.channels=C1a1.sinks.k1.channel= C1

This is set to memory, read the RAM, through the Telnet localhost 44444 port connection under the session information will be flume monitored.

Test

Four-start and test

Start Flume:

[[email protected] flume]# bin/flume-ng agent--conf conf--conf-file conf/flume-conf.properties--name A1-DFLUME.ROOT.L Ogger=info,console

Create another session

[[Email protected] ~]# telnet localhost 44444
-bash:telnet:command not found

[email protected] ~]# Yum Install

......

Installed:
Telnet.x86_64 1:0.17-48.el6

complete!

To view Flume connected sessions

This information is captured.

Indicates that the flume is functioning properly.

The above is just a simple test.

More plug-ins, multi-agent, and so on, and then continue.

Flume Installation and use