Flume collection Examples of several sources of mobile phone logs

Example 1: Type Avro, create a avro.conf for testing in the Conf of Flume, as follows:
A1.sources = R1
A1.sinks = K1
A1.channels = C1

# Describe/configure The source
A1.sources.r1.type = Avro
A1.sources.r1.channels = C1
A1.sources.r1.bind = 0.0.0.0
A1.sources.r1.port = 44444

# Describe The sink
A1.sinks.k1.type = Logger

# Use a channel which buffers events in memory
A1.channels.c1.type = Memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
A1.sources.r1.channels = C1
A1.sinks.k1.channel = C1

Start flume

/usr/flume/apache-flume-1.6. 0-bin/bin/flume-ngAgent--Conf/usr/flume/apache-flume-1.6. 0-bin/conf/ --Conf-file/usr/flume/apache-flume-1.6. 0-bin/conf/avro.Conf--Name A1-dflume.Root.Logger=Info,console

Open a window that is still outside, wear a test file, write some test information, and send it to Flume via Avro.

"hellow word"> /var/log/27.log

/usr/flume/apache-flume-1.6.0-bin/bin/flume-ng avro-client-c /usr/flume/apache-flume-1.6.0-bin/conf/-H-p44444-F /var/log/27.log

Results

The instance 2:spool monitors the configured directory for new files and reads the data from the file. Need to note two points: 1spool directory files can not be opened edit, 2spool directory cannot contain the corresponding subdirectories

Spool.conf

A1. Sources=R1A1. Sinks= K1A1. Channels= C1# describe/configure The sourceA1. Sources. R1. Type= Spooldira1. Sources. R1. Channels= C1A1. Sources. R1. Spooldir=/VAR/LOG/PPPA1. Sources. R1. Fileheader= True# Describe The sinkA1. Sinks. K1. Type= Logger# Use a channel which buffers events in memoryA1. Channels. C1. Type= Memorya1. Channels. C1. Capacity= +A1. Channels. C1. Transactioncapacity= -# Bind The source and sink to the channelA1. Sources. R1. Channels= C1A1. Sinks. K1. Channel= C1

Start flume

/usr/flume/apache-flume-1.6. 0-bin/bin/flume-ngAgent--Conf/usr/flume/apache-flume-1.6. 0-bin/conf/ --Conf-file/usr/flume/apache-flume-1.6. 0-bin/conf/spool.Conf--Name A1-dflume.Root.Logger=Info,console

The directory for monitoring is/VAR/LOG/PPP

"hellow word"> /var/log/ppp/3.log

Results

Instance 3:exec executes a given command to get the source of the output, if you want to use the tail command, the required file is large enough to see the output
Exec.conf

A1. Sources=R1A1. Sinks= K1A1. Channels= C1# describe/configure The source#a1. Sources.r1.type = exec#a1. sources.r1.channels = C1#a1. Sources.r1.command = tail-f/var/log/flume.logA1. Sources. R1. Type= Netcata1. Sources. R1. Bind= Localhosta1. Sources. R1. Port=44444# Describe The sinkA1. Sinks. K1. Type= Logger# Use a channel which buffers events in memoryA1. Channels. C1. Type= Memorya1. Channels. C1. Capacity= +A1. Channels. C1. Transactioncapacity= -# Bind The source and sink to the channelA1. Sources. R1. Channels= C1A1. Sinks. K1. Channel= C1

Start flume

/usr/flume/apache-flume-1.6. 0-bin/bin/flume-ngAgent--Conf/usr/flume/apache-flume-1.6. 0-bin/conf/ --Conf-file/usr/flume/apache-flume-1.6. 0-bin/conf/exec.Conf--Name A1-dflume.Root.Logger=Info,console

Write content to the file:

forin {1..100};doecho"exec tail$i" >> /var/log/flume.log;echo$i0.1;done

Results:

Instance 4:syslogtcp listening on TCP ports as a data source
Syslogtcp.conf

A1. Sources=R1A1. Sinks= K1A1. Channels= C1# describe/configure The sourceA1. Sources. R1. Type= SYSLOGTCPA1. Sources. R1. Port= theA1. Sources. R1. Host= Localhosta1. Sources. R1. Channels= C1# Describe The sinkA1. Sinks. K1. Type= Logger# Use a channel which buffers events in memoryA1. Channels. C1. Type= Memorya1. Channels. C1. Capacity= +A1. Channels. C1. Transactioncapacity= -# Bind The source and sink to the channelA1. Sources. R1. Channels= C1A1. Sinks. K1. Channel= C1

Start flume

/usr/flume/apache-flume-1.6. 0-bin/bin/flume-ngAgent--Conf/usr/flume/apache-flume-1.6. 0-bin/conf/ --Conf-file/usr/flume/apache-flume-1.6. 0-bin/conf/syslogtcp.Conf--Name A1-dflume.Root.Logger=Info,console

Send content toward port 80:

echo"hello idoall.org syslog"5140

Results:

Flume collection Examples of several sources of mobile phone logs