Log4j Direct output log to Flume

Log4j Direct output log to Flume

This jar is a tool class provided by the CDH release of Cloudera, which can be configured to output log4j logs directly to the flume for easy log acquisition.

In the CDH5.3.0 version, it is: Flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar

Directory is:/opt/cloudera/parcels/cdh/lib/flume-ng/tools/

Specific Use exampleslog4j Configuration (log4j.properties)

log4j.category.com.xxx=info,console,flumelog4j.appender.console= org.apache.log4j.consoleappenderlog4j.appender.console.target=system.outlog4j.appender.console.layout= Org.apache.log4j.patternlayoutlog4j.appender.console.layout.conversionpattern= "%d{yyyy-MM-dd HH:mm:ss}%p [%c:%L] -%m%nlog4j.appender.flume = Org.apache.flume.clients.log4jappender.Log4jAppenderlog4j.appender.flume.Hostname = Localhostlog4j.appender.flume.Port = 4444log4j.appender.flume.unsafemode = truelog4j.appender.flume.layout= ORG.APACHE.LOG4J.PATTERNLAYOUTLOG4J.APPENDER.FLUME.LAYOUT.CONVERSIONPATTERN=%D{YYYY-MM-DD HH:mm:ss}%p [%c:%L]-%m %n

Configure Classpath

Include Log4j.properties and Flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar in Classpath

Writing Java test classes and exporting them as Test.jar

import java.util.date;import org.apache.commons.logging.log;import  org.apache.commons.logging.logfactory;public class writelog {    protected  static final log logger = logfactory.getlog (Writelog.class);     public static void main (String[] args)  throws interruptedexception {         // TODO Auto-generated method stub         while  (True)  {             //  log output every two seconds current system timestamp              logger.info (New date (). GetTime ());             thread.sleep (;           )  try {                 throw new exception ("Exception msg ");            }             catch  (exception e)  {                 logger.error ("Error:"  +  E.getmessage ());            }         }    }}

Writing the flume agent configuration file

a1.sources = r1a1.sinks =  k1a1.channels = c1 # describe/configure the sourcea1.sources.r1.type =  avroa1.sources.r1.bind = localhosta1.sources.r1.port = 4444 # describe  the sinka1.sinks.k1.type = file_rolla1.sinks.k1.sink.directory = /data/soft/ Flume/tmpa1.sinks.k1.sink.rollinterval=86400a1.sinks.k1.sink.batchsize=100a1.sinks.k1.sink.serializer= Texta1.sinks.k1.sink.serializer.appendnewline = false # use a channel which  buffers events in memorya1.channels.c1.type = memorya1.channels.c1.capacity  = 1000a1.channels.c1.transactioncapacity = 1000 # bind the source  and sink to the channela1.sources.r1.channels = c1a1.sinks.k1.channel  = C1 

Run the program

Upload the relevant program to the server and start the agent first

After entering the Flume installation directory, perform

? Bin/flume-ng agent-c conf-f conf/avro.conf--name a1-dflume.root.logger=info,console?

Execute the test program

Java-classpath./:flume-ng-log4jappender-1.5.0-cdh5.3.0-jar-with-dependencies.jar:test.jar Com.xxx.WriteLog

Check running results

Tail-f/data/soft/flume/tmp/1436164166461-1

2015-07-06 14:51:36 Error [com.xxx.writelog:27]-error:exception msg2015-07-06 14:51:36 error [com.xxx.writelog:28]-er ror:stack2015-07-06 14:51:36 INFO [com.xxx.writelog:21]-14361654969752015-07-06 14:51:38 ERROR [com.xxx.WriteLog:27] -Error:exception msg2015-07-06 14:51:38 error [com.xxx.writelog:28]-error:stack2015-07-06 14:51:38 INFO [ COM.XXX.WRITELOG:21]-1436165498977

Log4j Direct output log to Flume