Single flume agent practice on-line post-outage process logging

a single-node flume deployment
1 Hadoop Preparation
Create the Flume directory in HDFs and assign permissions for the flume directory to flume users
HDFs Dfs-mkdir Flume
HDFs Dfs-chown-r Flume:flume/flume
2 flume-env.sh
Enter ${flume_home}/conf

CP flume-env.sh.template flume-env.sh
VI flume-env.sh, setting environment variables for JAVA
export java_home=/usr/java/jdk1.7.0_79

3 flume-conf.properties
In the configuration below, application and DIR are configured in Logback.xml by the application system

agent1.sources=source1 agent1.channels=channel1 Agent1.sinks=sink1  
Agent1.sources.source1.type=avro agent1.sources.source1.bind=0.0.0.0 agent1.sources.source1.port=44444 Agent1.sources.source1.channels=channel1 agent1.channels.channel1.type=memory agent1.channels.channel1.capacity= 10000 agent1.channels.channel1.transactioncapacity=1000 agent1.channels.channel1.keep-alive=30 Agent1.sinks.sink1 . Type=hdfs Agent1.sinks.sink1.channel=channel1 agent1.sinks.sink1.hdfs.path=hdfs://dashuju174:9000/flume/%{  
application}/%{dir}/%y%m%d Agent1.sinks.sink1.hdfs.filetype=datastream Agent1.sinks.sink1.hdfs.writeformat=text Agent1.sinks.sink1.hdfs.rollinterval=0 agent1.sinks.sink1.hdfs.rollsize=10240 agent1.sinks.sink1.hdfs.rollCount= 0 agent1.sinks.sink1.hdfs.idletimeout=60 

The following log is added to the

4 logback configuration
Logback.xml, specifically for the trace-level logs. The reason for selecting Trace is that developers are more familiar with info\error\debug and unfamiliar with trace applications, so developers are required to use trace-level logging uniformly.

<appender name= "flume" class= "Com.gilt.logback.flume.FlumeLogstashV1Appender" > <!--This is the IP and port connected to the Flume node--<flumeAgents> 192.168.5.174:44444 </flumeAgents> <FLUMEPROPERTIES&G T;connect-timeout=4000;request-timeout=8000</flumeproperties> <batchSize>100</batchSize> < Reportingwindow>1000</reportingwindow> <!--This configuration Avro header information--<additionalavroheaders>myheader= Dengjun</additionalavroheaders> <!--This configuration is currently applied--<application>bds</application> <filte R class= "Ch.qos.logback.classic.filter.LevelFilter" ><!--print error log only--<level>TRACE</level> < ;onmatch>accept</onmatch> <onMismatch>DENY</onMismatch> </filter> <layout class= "ch. Qos.logback.classic.PatternLayout "> <pattern>%message%n%ex</pattern> </layout> </appender 

5 environment variable
Flume relies on zookeeper, so you need to install Zookeeper
Tar zxvf apache-flume-1.6.0-bin.tar.gz before installing flume
Ln-s apache-flume-1.6.0 flume
Setting environment variables for flume
Export Flume_home=/home/hadoop/application/flume
Export PA Th= Path:path:flume_home/bin
two on-line running down record
Following the above configuration, the 3rd day of online, FLUME down. I have documented the settlement process here, after all, for the first time using flume.
to Be Continued ...