Data Collection with Apache Flume (ii)

Continue today to discuss the configuration of several agents.

The first agent is to capture the output of a particular command execution from the terminal and output the file to a specific directory. First look at the configured code:

Agent2.sources = Execsource      //Specifies the Sourceagent2.sinks = Filesink to get output from the command =//          output to file Sinkagent2.channels = FileChannel//     output to file Channelagent2.sources.execsource.type = exec  //Type Agent2.sources.execsource.command = Cat/home/leung/message   //specify command Agent2.sinks.filesink.type = FILE_ROLLagent2.sinks.filesink.sink.directory =/ Home/leung/flume/files     //output Directory agent2.sinks.filesink.sink.rollInterval = 0agent2.channels.filechannel.type = Fileagent2.channels.filechannel.checkpointDir =/home/leung/flume/fc/checkpoint// Checkpoint Agent2.channels.filechannel.dataDirs =/home/leung/flume/fc/data  // Data directory for Channel agent2.sources.execsource.channels = Filechannelagent2.sinks.filesink.channel = FileChannel

OK, start Agent2, and then view the results.

The results are as follows. As you can see, after executing the cat/home/leung/message command, the result of the output is consistent with the contents of the file in the files directory, proving that the file has been successfully written.

The next agent takes the data from the network port and writes it to HDFs in the Hadoop cluster. First look at the configuration code:

Agent4.sources = Netsource agent4.sinks = hdfssink//hdfs sinkagent4.channels = Memorychannelagent4.sources.netsource.type = Netcatagent4.sources.netsource.bind = Localhostagent4.sources.netsource.port = 3000agent4.sinks.hdfssink.type = Hdfsagent4.sinks.hdfssink.hdfs.path =/ Flume  //write out to the file directory on HDFs, do not need to create Agent4.sinks.hdfssink.hdfs.filePrefix = log  // Specifies the filename prefix of the write-out file Agent4.sinks.hdfssink.hdfs.rollInterval = 0agent4.sinks.hdfssink.hdfs.rollcount = 3agent4.sinks.hdfssink.hdfs.filetype = DataStreamagent4.channels.memorychannel.type = memoryagent4.channels.memorychannel.capacity = 1000agent4.channels.memorychannel.transactioncapacity = 100agent4.sources.netsource.channels = Memorychannelagent4.sinks.hdfssink.channel = Memorychannel

Start agent4 below and look at the results.

Here's a look at the results. found that a new flume folder has been created in HDFs and that the specified content has been written.

We then add a timestamp to the folder name. See the configuration code below for details.

agent5.sources = Netsourceagent5.sinks = Hdfssinkagent5.channels = Memorychannelagent5.sources.netsource.type = Netcatagent5.sources.netsource.bind = Localhostagent5.sources.netsource.port = 3000agent5.sources.netsource.interceptors = Tsagent5.sources.netsource.interceptors.ts.type = Org.apache.flume.interceptor.timestampinterceptor$builder//Refer to this class method to add a timestamp Agent5.sinks.hdfssink.type = Hdfsagent5.sinks.hdfssink.hdfs.path =/flume-%y-%m-%d//define folder name format Agent5.sinks.hdfssink.hdfs.filePrefix = Log-agent5.sinks.hdfssink.hdfs.rollinterval = 0agent5.sinks.hdfssink.hdfs.rollcount = 3agent5.sinks.hdfssink.hdfs.filetype = DataStreamagent5.channels.memorychannel.type = memoryagent5.channels.memorychannel.capacity = 1000agent5.channels.memorychannel.transactioncapacity = 100agent5.sources.netsource.channels = Memorychannelagent5.sinks.hdfssink.channel = Memorychannel

OK, start agent5 below.

Here's a look at the results. You can see that the name of the folder was added to the date as expected.

OK, first here, there are two slightly more complicated agents to discuss next time. My level is limited, please do not hesitate to correct me! Thank you!

Data Collection with Apache Flume (ii)