"OD Big Data Combat" flume combat

First, Netcat source + memory Channel + logger SINK1. Modify Configuration

1) Modify the flume-env.sh file under $flume_home/conf, modify the contents as follows

Export JAVA_HOME=/OPT/MODULES/JDK1. 7. 0_67

2) under the $flume_home/conf directory, create the agent subdirectory, creating a new netcat-memory-logger.conf with the following configuration:

# netcat-memory-logger# Name The components in this agenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the Sourcea1.sources.r1.type=Netcata1.sources.r1.bind= beifeng-hadoop- GenevaA1.sources.r1.port=44444# Describe The Sinka1.sinks.k1.type=logger# use a channelwhichBuffers EventsinchMemorya1.channels.c1.type=memorya1.channels.c1.capacity= +a1.channels.c1.transactionCapacity= -# Bind The source and sink to the Channela1.sources.r1.channels=C1a1.sinks.k1.channel= C1

2. Start Flume and test

1) Start

Bin/flume-ng agent-n a1-c conf/-F Conf/agent/netcat-memory-logger.conf-dflume.root.logger=info,console

2) Testing

NC beifeng-hadoop-44444

Enter any string and observe the server's log file.

Use the NC command of Linux and install it first if the command does not exist.

Installing Netcat:sudo yum-y Install NC

Second, Agent:avro source + file Channel + HDFs sink 

1. Add Configuration

Under the $flume_home/conf directory, create the agent subdirectory, creating a new avro-file-hdfs.conf with the following configuration:

# Name The components in this agenta1.sources=r1a1.sinks=K1a1.channels=c1# Describe/Configure the Sourcea1.sources.r1.type=Netcata1.sources.r1.bind= beifeng-hadoop- GenevaA1.sources.r1.port=4141# Describe The Sinka1.sinks.k1.type=Hdfsa1.sinks.k1.hdfs.path= HDFs://beifeng-hadoop-02:9000/flume/events/%y-%m-%d# Default:FlumeDataa1.sinks.k1.hdfs.filePrefix=FlumeDataa1.sinks.k1.hdfs.useLocalTimeStamp=trueA1.sinks.k1.hdfs.rollInterval=0A1.sinks.k1.hdfs.rollCount=0# generally close to block -  -  thea1.sinks.k1.hdfs.rollSize=10240A1.sinks.k1.hdfs.fileType=Datastream#a1.sinks.k1.hdfs.round=true#a1. Sinks.k1.hdfs.roundValue=Ten#a1. Sinks.k1.hdfs.roundUnit=minute# use a channelwhichBuffers EventsinchMemorya1.channels.c1.type=fileA1.channels.c1.checkpointDir=/opt/modules/cdh/apache-flume-1.5.0-cdh5.3.6-bin/Checkpointa1.channels.c1.dataDirs=/opt/modules/cdh/apache-flume-1.5.0-cdh5.3.6-bin/data# Bind The source and sink to the Channela1.sources.r1.channels=C1a1.sinks.k1.channel= C1

2. Start and test

1) Start Flume agent

Bin/flume-ng agent-n a1-c conf/-F conf/agent/avro-file-hdfs.conf-dflume.root.logger=info,console

2) avro-client test with Flume

Bin/flume-ng avro-client--host beifeng-hadoop-4141 --filename/home/beifeng/order_info.txt

"OD Big Data Combat" flume combat