Flume of OD Studies 20160806

First, Flume

Flume is a distributed, reliable, usable, and very efficient service for collecting, aggregating, and moving information about large volumes of log data.

1. How to Structure

1) All applications use one flume server;

2) All applications share flume cluster;

3) Each application uses one flume, and then uses a flume node to collect scattered flume data;

2. Flume Components

1) Each flume process (JVM process) that is started, called the agent

Each flume agent:

Source: Combines with an external data source to obtain or receive data

(1) Source actively fetches data from an external data source

(2) Source receives data sent from an external data source.

Channel: Cache, similar queue, FIFO

(1) Fault tolerance: File Channel

(2) Buffer

Sink: Get data from the channel and send it out

Flume: Simply select the appropriate source, channel, sink type

II. Installation and Deployment

1) Download the installation package

2) Unzip the installation

3) Modify the configuration file

Flume-env.sh.template-Flume-env.sh.template

Java_home

Java_opts

Flume of OD Studies 20160806