Construction of Flume Service

Unify the time before building, turn off the firewall, use the jar package version is 1.6.0

There are two ways to configure a service

The first type: The following steps:

1. Pass the jar package to the Node1 and extract it to the root directory

2. Change the directory name by using the following command: MV apache-flume-1.6.0-bin/home/install/flume-1.6

3. After entering the flume-1.6 directory, vi test1, create test1 file, open

https://flume.apache.org/FlumeUserGuide.html link, copy the piece of code about source, channel, and sink

Change one of the localhost to Node1

4. Running Flume

5. Open Node2, enter command: Telnet node1 44444

Any word such as Hello, see if Node1 has received the Hello character

CTRL + ", quit quit, CTRL + C terminate

The second way: The steps are as follows:

1. Start the Zookeeper service first: zkserver.sh start, then dfs:start-dfs.sh

2.VI test2

Specific code:

[Root@node1 flume-1.6]# VI test2

A1.sources = R1
A1.sinks = K1
A1.channels = C1

# Describe/configure The source
A1.sources.r1.type =spooldir
A1.sources.r1.spooldir=/opt/flume

# Describe The sink
A1.sinks.k1.type =avro
A1.sinks.k1.hostname=node2
a1.sinks.k1.port=55555

# Use a channel which buffers events in memory
A1.channels.c1.type = Memory
A1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
A1.sources.r1.channels = C1
A1.sinks.k1.channel = C1

On Node2 (equivalent to server)

Specific code:
A1.sources = R1
A1.sinks = K1
A1.channels = C1

# Describe/configure The source
A1.sources.r1.type =avro
A1.sources.r1.bind=node2
a1.sources.r1.port=55555

# Describe The sink
A1.sinks.k1.type =hdfs
a1.sinks.k1.hdfs.path=hdfs://xiaotian/flume/%y-%m-%d/
A1.sinks.k1.hdfs.fileprefix=%h%m
A1.sinks.k1.hdfs.rollsize=0
A1.sinks.k1.hdfs.rollcount=0
A1.sinks.k1.hdfs.rollinterval=60
A1.sinks.k1.hdfs.idletimeout=1
A1.sinks.k1.hdfs.filetype=datastream
A1.sinks.k1.hdfs.round=true
A1.sinks.k1.hdfs.roundvalue=1
A1.sinks.k1.hdfs.roundunit=minute
A1.sinks.k1.hdfs.uselocaltimestamp=true

# Use a channel which buffers events in memory
A1.channels.c1.type = Memory
A1.channels.c1.capacity = 100000
a1.channels.c1.transactionCapacity = 1000

# Bind the source and sink to the channel
A1.sources.r1.channels = C1
A1.sinks.k1.channel = C1
~

3. Enter the Conf directory in the flume-1.6

MV Flume-env.sh.template flume-env.sh

VI flume-env.sh

Add a sentence: java_opts= "-xms100m-xmx300m"

4. Create a new/opt/flume directory, and then start the flume, start the Node2 (change source), and then start the Node1

Bin/flume-ng agent--conf conf--conf-file test2--name A1-dflume.root.logger=info,console

After you start Node1, you can see the process on node first to see if it started successfully: NETSTAT-NTPL

Construction of Flume Service