Flume ng installation Deployment and data acquisition testing

Reprint Please specify source: http://www.cnblogs.com/xiaodf/

Flume as a Log collection tool, monitoring a file directory or a file, when new data is added, the acquisition of new data sent to the message queue.

1 Installing the Deployment flume

To collect local data from a data node, each node needs to have a flume tool installed to do data collection.

1.1 Download and install

Go to the official website to download the latest version of Flume

As: http://flume.apache.org/, currently the latest version is 1.6.0, requires 1.7 and later versions of the JDK.

1. Decompression

TAR-XZVF apache-flume-1.6.0-bin.tar.gz-c/usr/local/

2, Installation JDK1.7

If the JDK version on the node is less than 1.7, you need to install the JDK version 1.7 or later

JDK 1.7:

Http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

Create a Java directory under the Flume directory to store the JDK

Cd/usr/local/apache-flume-1.6.0-binmkdir JAVACD JAVATAR-XZVF jdk-7u79-linux-x64.tar.gz

1.2 Configuring Flume System parameters

Modify the flume-env.sh configuration file, mainly the Java_home variable settings

CD/USR/LOCAL/APACHE-FLUME-1.6.0-BIN/CONFCP flume-env.sh.template flume-env.sh

Set the Flume_classpath variable and the java_home variable inside the flume-env.sh,

Example:

Export Java_home=/usr/local/apache-flume-1.6.0-bin/java/jdk1.7.0_79flume_classpath= "/usr/local/ Apache-flume-1.6.0-bin/"

Variable specific content based on actual modification

1.3 Adding flume third-party dependencies

A flume interceptor has been customized here, so you can ignore this step.

Add a third-party dependency package Flume-plugins-1.0-snapshot.jar, this package implements a flume interceptor, the data collected by the flume is serialized, structured, and so on, and finally each data generated an event data return.

CD/USR/LOCAL/APACHE-FLUME-1.6.0-BINMKDIRPLUGINS.D    -Create dependent directory, directory name must be Plugins.dcdplugins.dmkdirflume-plugins          --project directory, directory name arbitrarily cdflume-pluginsmkdirlib           --jar package directory, directory name must be Lib

Place the third-party jar package Flume-plugins-1.0-snapshot.jar in the Lib directory

1.4 Creating the Flume agent configuration file

Create a flume boot configuration file that specifies the contents of the Source,channel,sink 3 components. There are several configuration options for each component, please see the flume website for details.

Create the configuration file flume.conf, with the following examples:

a1.sources = X1a1.sinks = Y1a1.channels = z1# describe/configure Thesourcea1.sources.x1.type = Execa1.sources.x1.channels =z1a1.sources.x1.command =tail-f/home/xdf/exec.txt# Describe the Sinka1.sinks.y1.type = logger# use a channel whichbuffers events in Memorya1.channels.z1.type =memorya1.channels.z1.capacity= 1000a1.channels.z1.transactioncapacity= 100# Bind The source andsink to the Channela1.sources.x1.channels = Z1a1.sinks.y1.channel = Z1

1.5 Starting the Flume Agent

Enter the flume installation directory to open the agent.

Cd/usr/local/apache-flume-1.6.0-bin./bin/flume-ng agent--conf conf--conf-file flume.conf--name A3- Dflume.root.logger=info,console

Note:-dflume.root.logger=info,console is only used for debug, do not production environment mechanically, otherwise a large number of logs will be returned to the terminal. -c/--conf followed by the configuration directory,-f/--conf-file followed by the specific configuration file,-n/--name the name of the specified agent.

1.6 Testing

The above configuration of the example.conf file, the implementation of the function is to monitor the file/home/xdf/exec.txt,

If new data is written, Flume will collect new data and print it on the console.

Test Case: Writes the content "Hello Flume" to the/home/xdf/exec.txt file.

echo ' Hello Flume ' >>/home/xdf/exec.txt

The Flume terminal window will now print the following message, indicating success:

2015-06-3016:01:52,910 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.LoggerSink.process (loggersink.java:94)] Event: {headers:{} body:68 6C 6C 6F (6C) 6D + He Llo Flume}

At this point, the flume installation is complete.

Flume ng installation Deployment and data acquisition testing