Flume collecting logs, writing to HDFs

Last Update:2016-12-06 Source: Internet

Author: User

Developer on Alibaba Coud: Build your first app with APIs, SDKs, and tutorials on the Alibaba Cloud. Read more ＞

First install the Flume:

It is recommended to maintain a unified user with Hadoop to install Hadoop,flume

This time I use Hadoop user installation flume

http://douya.blog.51cto.com/6173221/1860390

To start the configuration:

1, configuration file Writing:

Vim flume_hdfs.conf

# Define A memory channel called CH1 on Agent1

Agent1.channels.ch1.type = Memory

Agent1.channels.ch1.capacity = 10000

agent1.channels.ch1.transactionCapacity = 100

#agent1. channels.ch1.keep-alive = 30

#define Source Monitor A file

Agent1.sources.avro-source1.type = Exec

Agent1.sources.avro-source1.shell =/bin/bash-c

Agent1.sources.avro-source1.command = Tail-n +0-f/root/logs/appcenter.log

Agent1.sources.avro-source1.channels = Ch1

Agent1.sources.avro-source1.threads = 5

Agent1.sources.avro-source1.interceptors = I1

Agent1.sources.avro-source1.interceptors.i1.type = Org.apache.flume.interceptor.timestampinterceptor$builder

# Define A logger sink that simply logs all events it receives

# and connect it to the other end of the same channel.

Agent1.sinks.log-sink1.channel = Ch1

Agent1.sinks.log-sink1.type = HDFs

Agent1.sinks.log-sink1.hdfs.path = hdfs://ns1/flume/%y%m%d

Agent1.sinks.log-sink1.hdfs.writeformat = events-

Agent1.sinks.log-sink1.hdfs.filetype = DataStream

Agent1.sinks.log-sink1.hdfs.rollinterval = 60

Agent1.sinks.log-sink1.hdfs.rollsize = 134217728

Agent1.sinks.log-sink1.hdfs.rollcount = 0

#agent1. sinks.log-sink1.hdfs.batchsize = 100000

#agent1. Sinks.log-sink1.hdfs.txneventmax = 100000

#agent1. Sinks.log-sink1.hdfs.calltimeout = 60000

#agent1. Sinks.log-sink1.hdfs.appendtimeout = 60000

# Finally, Now that we've defined all the components, tell

# agent1 which ones we want to Activate.

Agent1.channels = Ch1

Agent1.sources = Avro-source1

Agent1.sinks = Log-sink1

2, provided that the Hadoop cluster starts normally and is available,

3, start flume, Start collecting logs

Start:

Flume-ng agent-c/usr/local/elk/apache-flume/conf/-f/usr/local/elk/apache-flume/conf/flume_hdfs.conf- Dflume.root.logger=info,console-n Agent1

Error one:

2016-12-06 11:24:49,036 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.hdfs.BucketWriter.open (bucketwriter.java:234)] Creating hdfs://ns1/flume// Flumedata.1480994688831.tmp

2016-12-06 11:24:49,190 (sinkrunner-pollingrunner-defaultsinkprocessor) [ERROR- Org.apache.flume.sink.hdfs.HDFSEventSink.process (hdfseventsink.java:459)] Process failed

Java.lang.noclassdeffounderror:org/apache/hadoop/util/platformname

At Org.apache.hadoop.security.UserGroupInformation.getOSLoginModuleName (usergroupinformation.java:366)

At Org.apache.hadoop.security.usergroupinformation.<clinit> (usergroupinformation.java:411)

At Org.apache.hadoop.fs.filesystem$cache$key.<init> (FILESYSTEM.JAVA:2828)

At Org.apache.hadoop.fs.filesystem$cache$key.<init> (filesystem.java:2818)

At Org.apache.hadoop.fs.filesystem$cache.get (filesystem.java:2684)

At Org.apache.hadoop.fs.FileSystem.get (filesystem.java:373)

At Org.apache.hadoop.fs.Path.getFileSystem (path.java:295)

At Org.apache.flume.sink.hdfs.bucketwriter$1.call (bucketwriter.java:243)

At Org.apache.flume.sink.hdfs.bucketwriter$1.call (bucketwriter.java:235)

At Org.apache.flume.sink.hdfs.bucketwriter$9$1.run (bucketwriter.java:679)

At Org.apache.flume.auth.SimpleAuthenticator.execute (simpleauthenticator.java:50)

At Org.apache.flume.sink.hdfs.bucketwriter$9.call (bucketwriter.java:676)

At Java.util.concurrent.FutureTask.run (futuretask.java:266)

At Java.util.concurrent.ThreadPoolExecutor.runWorker (threadpoolexecutor.java:1142)

At Java.util.concurrent.threadpoolexecutor$worker.run (threadpoolexecutor.java:617)

At Java.lang.Thread.run (thread.java:745)

caused By:java.lang.ClassNotFoundException:org.apache.hadoop.util.PlatformName

At Java.net.URLClassLoader.findClass (urlclassloader.java:381)

At Java.lang.ClassLoader.loadClass (classloader.java:424)

At Sun.misc.launcher$appclassloader.loadclass (launcher.java:331)

At Java.lang.ClassLoader.loadClass (classloader.java:357)

... More

Solve:

1,cd/usr/local/hadoop/hadoop/share/hadoop/common/

CP Hadoop-common-2.7.3.jar

2, Cd/usr/local/hadoop/hadoop/share/hadoop/common/lib

Hadoop-auth-2.7.3.jar

Copy this jar package from the Lib directory to the Flume client's Lib directory

But there will be a lot of errors, so lazy practice, the lib under all the jar package copy to the Flume side of the Lib

Reason: because flume to write data to the person hdfs, so need to rely on some HDFS package

Start:

Flume-ng agent-c/usr/local/elk/apache-flume/conf/-f/usr/local/elk/apache-flume/conf/flume_hdfs.conf- Dflume.root.logger=info,console-n Agent1

Error two:

2016-12-06 11:36:52,791 (sinkrunner-pollingrunner-defaultsinkprocessor) [INFO- Org.apache.flume.sink.hdfs.BucketWriter.open (bucketwriter.java:234)] Creating hdfs://ns1/flume// Flumedata.1480995177465.tmp

2016-12-06 11:36:52,793 (sinkrunner-pollingrunner-defaultsinkprocessor) [WARN- Org.apache.flume.sink.hdfs.HDFSEventSink.process (hdfseventsink.java:455)] HDFs IO Error

Java.io.IOException:No FileSystem for Scheme:hdfs

At Org.apache.hadoop.fs.FileSystem.getFileSystemClass (filesystem.java:2660)

At Org.apache.hadoop.fs.FileSystem.createFileSystem (filesystem.java:2667)

At org.apache.hadoop.fs.filesystem.access$200 (filesystem.java:94)

At Org.apache.hadoop.fs.filesystem$cache.getinternal (filesystem.java:2703)

At Org.apache.hadoop.fs.filesystem$cache.get (filesystem.java:2685)