Flume principle, installation and use

1.flume is a distributed log collection system that transmits collected data to its destination. 2.flume has a core concept, called an agent. The agent is a Java process that runs on the Log collection node. The 3.agent consists of 3 core components: source, channel, sink. The 3.1 source component is dedicated to collecting logs and can handle various types of log data in various formats, including Avro, thrift, exec, JMS, spooling directory, netcat, sequence generator, Syslog, HTTP, Legacy, custom. The     Source component collects the data and temporarily stores it in the channel. The 3.2 channel component is used by the agent to temporarily store data, which can be stored in memory, JDBC, file, and custom. The data in the     channel will not be deleted until the sink is sent successfully. The 3.3 sink component is a component used to send data to a destination, including HDFs, logger, Avro, thrift, IPC, file, NULL, HBase, SOLR, and custom. 4. During the entire data transfer process, the event is flowing. The transaction guarantee is at the event level. 5.flume can support multi-level flume agent, support fan-in (fan-in), fan-out (fan-out).  6. Writing configuration file example  #agent1表示代理名称agent1 .sources=source1agent1.sinks=sink1agent1.channels=channel1    #Spooling directory is to monitor changes to new files in the specified folder, parse the contents of the file as soon as the new file appears, and then write to Channle. When the write is complete, mark the file as completed or delete the file. #配置source1agent1. sources.source1.type=spooldiragent1.sources.source1.spooldir=/root/ Hmbbsagent1.sources.source1.channels=channel1agent1.sources.source1.fileheader = Falseagent1.sources.source1.interceptors = i1agent1.sources.soUrce1.interceptors.i1.type = timestamp  #配置sink1agent1. sinks.sink1.type=hdfsagent1.sinks.sink1.hdfs.path= hdfs://hadoop0:9000/hmbbsagent1.sinks.sink1.hdfs.filetype=datastreamagent1.sinks.sink1.hdfs.writeformat= Textagent1.sinks.sink1.hdfs.rollinterval=1agent1.sinks.sink1.channel= channel1agent1.sinks.sink1.hdfs.fileprefix=%y-%m-%d  #配置channel1agent1. channels.channel1.type= Fileagent1.channels.channel1.checkpointdir=/root/hmbbs_tmp/123agent1.channels.channel1.datadirs=/root/hmbbs_ Tmp/  7. Execute command bin/flume-ng agent-n agent1-c conf-f conf/example-dflume.root.logger=debug,console

Flume principles, installation, and use