Flume netcat Passive Source Example

A flume task is an agent that consists of three parts, as shown in the figure:

Mainly focus on source and sink.

Source is divided into active source and passive source.

Sink such as HDFs client, Kafka client, etc.

TAR-ZXVF apache-flume-1.6.0-bin.tar.gz

Configure Environment variables

Vim ~/.bash_profile

...

Source ~/.bash_profile

Vim test01

# example.conf:a Single-node Flume Configuration

# Name The components on this agent
a1.sources = R1
a1.sinks = K1
a1.channels = C1

# describe/configure the source
a1.sources.r1.type = netcat A1.sources.r1.bind
= n Ode
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = Logger

# Use a channel which buffer s events in memory
a1.channels.c1.type = Memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity =

# Bind The source and sink to the channel
a1.sources.r1.channels = c1
  a1.sinks.k1.channel = C1

$ bin/flume-ng Agent--conf apache-flume-1.6.0-bin/conf/--conf-file test01--name

apache-flume-1.6.0-bin/conf/is the flume configuration file.

TEST01 is the configuration file for the current agent.

It is a TCP protocol that can be viewed through the netstat-ntpl command

Now send a message via Telnet (TCP protocol Client):

Command telnet node1:44444

Then you can send a message in the shell.

(View yum source More/etc/yum.repos.d/local.repo

Install Telnet via yum,yum-y install telnet)

Official website : https://flume.apache.org/FlumeUserGuide.html