Flume installation Configuration

Tagged: NET ogg local port Javah Port data event multiple

1 Download and unzip the installation package

: http://flume.apache.org/download.html

Decompression: Tar zxvf apache-flume-1.8.0-bin.tar.gz

2 Configuring Environment variables

VI ~/.BASHRC

To configure environment variables:

Export Flume_home=/hmaster/flume/apache-flume-1.8.0-bin
Export flume_conf_dir= $FLUME _home/con

Export path= $PATH: $FLUME _home/bin

Let the configuration take effect

Source: Bashrc

Configuring Javahome for flume-env.sh files

Export java_home=/hmaster/javaenv/jdk1.8.0_181

3 Flume Deployment Example 1 Avro

Create a new netcat.conf configuration file in the/hmaster/flume/apache-flume-1.8.0-bin/conf directory

Flume can listen to a port and capture the transferred data through Avro, with the following configuration examples:

#那么我们也给这个三个组件分别取名字
A2.sources = R1
A2.channels = C1
A2.sinks = K1

#定义具体的source内容
#这里是执行命令以及下面对应的具体命令
#这个命令执行后的数据返回给这个source
A1.sources.r1.type = Netcat

A1.sources.r1.bind = localhost

A1.sources.r1.port = 44444

#定义具体的channel信息
#我们source定义好了, we need to define our channel.
A2.channels.c1.type = Memory
A2.channels.c1.capacity = 10000
a2.channels.c1.transactionCapacity = 100

#定义具体的sink信息
#这个logger sink is to print the information directly to the console
#就是打印日志
A2.sinks.k1.type = Logger

#最后来组装我们之前定义的channel和sink
A2.sources.r1.channels = C1
A2.sinks.k1.channel = C1

Run Flumeagent, listen for 44444 ports on this machine

-N corresponds to the agent name

-C Configuration file directory

-F Configuration file address

Flume-ng agent-n a1-c conf-f. /conf/netcat.conf

Open another terminal, log in to localhost by Telnet 44444, enter test data

$ telnet localhost 44444

View Flume Data collection situation

2 Spool

1 Configure spool.conf to monitor directory userlogs files, send file contents to local 60000 port

Spool is used to monitor the configuration of new files in the directory and to read the data in the file. Two points to note: Files copied to the spool directory can no longer be opened for editing, and the spool directory cannot contain the corresponding subdirectories. Examples of specific configuration files such as the following

#定义agent名称, the name of the Source,channel,sink
#a1就是我们给agent起的名字, we know that there are multiple agents, and that's how we make the difference.
#我们知道agent包含了三个重要的组件, there's source,channel,sink.
#那么我们也给这个三个组件分别取名字
A2.sources = R1
A2.channels = C1
A2.sinks = K1

#定义具体的source内容
#这里是执行命令以及下面对应的具体命令
#这个命令执行后的数据返回给这个source
A2.sources.r1.type = Spooldir
A2.sources.r1.spoolDir =/home/hadoop/hadoop-2.9.0/userlogs

#定义具体的channel信息
#我们source定义好了, we need to define our channel.
A2.channels.c1.type = Memory
A2.channels.c1.capacity = 10000
a2.channels.c1.transactionCapacity = 100

#定义具体的sink信息
#就是将数据转换成Avro Event is then sent to the configured RPC port
A2.sinks.k1.type = Avro
A2.sinks.k1.hostname= localhost
a2.sinks.k1.port= 60000

#最后来组装我们之前定义的channel和sink
A2.sources.r1.channels = C1
A2.sinks.k1.channel = C1

2 configuration spool2.conf for fetching data from local 60000 ports and writing to HDFs

#定义agent名称, the name of the Source,channel,sink
A3.sources = R1
A3.channels = C1
A3.sinks = K1

#定义具体的source内容
A3.sources.r1.type= Avro
a3.sources.r1.bind= localhost
a3.sources.r1.port= 60000
#定义具体的channel信息
#我们source定义好了, we need to define our channel.
A3.channels.c1.type = Memory
A3.channels.c1.capacity = 10000
a3.channels.c1.transactionCapacity = 100

#定义具体的sink信息
A3.sinks.k1.type = HDFs
A3.sinks.k1.hdfs.path = Hdfs://192.168.79.2:9000/flume/event2
A3.sinks.k1.hdfs.filePrefix = events-
A3.sinks.k1.hdfs.fileType = DataStream

#最后来组装我们之前定义的channel和sink
A3.sources.r1.channels = C1
A3.sinks.k1.channel = C1

Flume installation Configuration