Flume collection Examples of several sources for collecting logs

Example 1: Type Avro, create a avro.conf for testing in the Conf of Flume, as follows:
A1.sources = R1
A1.sinks = K1
A1.channels = C1

# Describe/configure The source
A1.sources.r1.type = Avro
A1.sources.r1.channels = C1
A1.sources.r1.bind = 0.0.0.0
A1.sources.r1.port = 44444

# Describe The sink
A1.sinks.k1.type = Logger

# Use a channel which buffers events in memory
A1.channels.c1.type = Memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
A1.sources.r1.channels = C1
A1.sinks.k1.channel = C1

Start flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng Agent--conf/usr/flume/apache-flume-1.6.0-bin/conf/--conf-file/ usr/flume/apache-flume-1.6.0-bin/conf/avro.conf--name A1-dflume.root.logger=info,console

Open a window that is still outside, wear a test file, write some test information, and send it to Flume via Avro.

echo "Hellow word" >/var/log/27.log

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng avro-client-c/usr/flume/apache-flume-1.6.0-bin/conf/-h localhost- P 44444-f/var/log/27.log

The result is as shown:

The instance 2:spool monitors the configured directory for new files and reads the data from the file. Need to note two points: 1spool directory files can not be opened edit, 2spool directory cannot contain the corresponding subdirectories

Spool.conf

a1.sources = R1
a1.sinks = K1
a1.channels = C1
# describe/configure the source
A1.sources.r1.type = Spool Dir
a1.sources.r1.channels = C1
A1.sources.r1.spoolDir =/var/log/ppp
A1.sources.r1.fileHeader = True
# Describe The sink
a1.sinks.k1.type = Logger
# Use a channel which buffers events in memory
A1.channels . C1.type = Memory
a1.channels.c1.capacity =
a1.channels.c1.transactionCapacity =
# Bind the Source and sink to the channel
a1.sources.r1.channels = C1
A1.sinks.k1.channel = C1

Start flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng Agent--conf/usr/flume/apache-flume-1.6.0-bin/conf/--conf-file/ usr/flume/apache-flume-1.6.0-bin/conf/spool.conf--name A1-dflume.root.logger=info,console

The directory for monitoring is/VAR/LOG/PPP

echo "Hellow word" >/var/log/ppp/3.log

Results

Instance 3:exec executes a given command to get the source of the output, if you want to use the tail command, the required file is large enough to see the output
Exec.conf

a1.sources = R1
a1.sinks = K1
a1.channels = C1
# describe/configure the source
#a1. Sources.r1.type = Exec
#a1. sources.r1.channels = C1
#a1. Sources.r1.command = tail-f/var/log/flume.log

a1.sources.r1.type = Netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444


# Describe the sink
A1.sinks.k1.type = Logger
# Use a channel which buffers events in memory
a1.channels.c1.type = Memory
a1.ch annels.c1.capacity =
a1.channels.c1.transactionCapacity = $
# Bind The source and sink to the CHANNEL
  a1.sources.r1.channels = C1
A1.sinks.k1.channel = C1

Start flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng Agent--conf/usr/flume/apache-flume-1.6.0-bin/conf/--conf-file/ usr/flume/apache-flume-1.6.0-bin/conf/exec.conf--name A1-dflume.root.logger=info,console

Write content to the file:

For i in {1..100};d o echo "exec tail$i" >>/var/log/flume.log;echo $i; Sleep 0.1;done

Results:

Instance 4:syslogtcp listening on TCP ports as a data source
Syslogtcp.conf

a1.sources = R1
a1.sinks = K1
a1.channels = C1
# describe/configure the source
a1.sources.r1.type = Syslo Gtcp
a1.sources.r1.port =
a1.sources.r1.host = localhost
a1.sources.r1.channels = C1
# Describe the Sink
A1.sinks.k1.type = Logger
# Use a channel which buffers events in memory
A1.channels.c1.type = Memory
  a1.channels.c1.capacity =
a1.channels.c1.transactionCapacity = $
# Bind The source and sink to the CHA Nnel
a1.sources.r1.channels = C1
A1.sinks.k1.channel = C1

Start flume

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng Agent--conf/usr/flume/apache-flume-1.6.0-bin/conf/--conf-file/ usr/flume/apache-flume-1.6.0-bin/conf/syslogtcp.conf--name A1-dflume.root.logger=info,console

Send content toward port 80:

echo "Hello idoall.org syslog" | NC localhost 5140

Results: