Construction of fault-tolerant environment of "Flume" Flume failover

There are many examples of failover on the Internet, but there are multiple approaches, and individuals feel that the principle of single responsibility

1, a machine running a flume agent

2, a agent downstream sink point to a flume agent, do not have a flume agent configuration multiple Ports "impact performance"

3, sub-machine configuration, you can avoid a driver, another can still be used, or accompany on a machine through the port to distinguish, once the crash, the overall collapse

Here's a look at the specific example:

The first is the configuration of the Flumet agent client

The higher the priority, the higher the precedence, and the sink will be used first.

# Name The components in this agenta1.sources = R1a1.sinks = K1 k2a1.channels = C1 # describe/configure the SOURCEA1.SOURC Es.r1.type = Execa1.sources.r1.channels=c1a1.sources.r1.command=tail-f/root/dev/biz/logs/bizlogic.log#define Sinkgroupsa1.sinkgroups=g1a1.sinkgroups.g1.sinks=k1 k2a1.sinkgroups.g1.processor.type= Failovera1.sinkgroups.g1.processor.priority.k1=10a1.sinkgroups.g1.processor.priority.k2= 5a1.sinkgroups.g1.processor.maxpenalty=10000#define the sink 1a1.sinks.k1.type=avroa1.sinks.k1.hostname= 192.168.11.179a1.sinks.k1.port=9876#define the sink 2a1.sinks.k2.type=avroa1.sinks.k2.hostname= 192.168.11.178a1.sinks.k2.port=9876# use a channel which buffers events in Memorya1.channels.c1.type = Memorya1.channels  . c1.capacity = 1000a1.channels.c1.transactioncapacity = 100# Bind The source and sink to the Channela1.sources.r1.channels = C1a1.sinks.k1.channel = C1a1.sinks.k2.channel=c1

Here you can see the use of Sinkgroup, which includes two sink, two sink pointing to different flume agents

Then look at the configuration of the Flume agent server, which is 179,178, to see a

# Name The components in this agenta1.sources = R1a1.sinks = K1a1.channels = C1 # describe/configure the sourcea1.sources. R1.type=avro#any Address to listena1.sources.r1.bind=0.0.0.0a1.sources.r1.port=9876a1.sources.r1.channels=c1# Describe the Sinka1.sinks.k1.type = file_rolla1.sinks.k1.sink.directory=/root/dev/flumeout/ filea1.sinks.k1.sink.rollinterval=3600# use a channel which buffers events in Memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactioncapacity = 100# Bind The source and sink to the channela1.so Urces.r1.channels = C1a1.sinks.k1.channel = C1

It can be seen that the Flume agent client and server between the Avro to transfer data, Avro is flume built-in protocol, very convenient, can be flume whole string up

Next, start the Flume agent server, and then start the Flume agent client

The test is as follows:

For i in {1..100};d o echo "exec test tail-f $i on  Terminator 176" >> Bizlogic.log;echo $i; Sleep 0.1;done

To write content to the file, triggering the Flume agent client tail-f, so that the content will be through the Flume agent client into the memory channel, in the failover mechanism to select the high priority sink to output, the final output of the place, There is a final ring in the flume configuration Sink.type decision, you can see is file_roll, that is, the file form written to disk, will be scrolled in a certain way

At first start, 178 and 179 will produce this file, but when you start producing the file content, there are still 179 to write the file content.

At this point, the complete flume failover mechanism has gone through, mutual encouragement!

Construction of fault-tolerant environment of "Flume" Flume failover