"Flume" Flume load Balancing Environment construction Load_balance

Flume load Balancing is the choice of a certain algorithm per sink output to the specified place, if the file output is very large, load balancing is still necessary, through the output of multiple channels to alleviate the output pressure

Flume built-in load balancing algorithm by default is round robin, polling algorithm, ordered selection

Here's a look at the specific examples:

# Name The components in this agenta1.sources = R1a1.sinks = K1 k2a1.channels = C1 # describe/configure the SOURCEA1.SOURC Es.r1.type = execa1.sources.r1.channels=c1a1.sources.r1.command=tail-f/root/dev/biz/logs/bizlogic.log #define Sinkgroupsa1.sinkgroups=g1a1.sinkgroups.g1.sinks=k1 K2a1.sinkgroups.g1.processor.type=load_ Balancea1.sinkgroups.g1.processor.backoff=truea1.sinkgroups.g1.processor.selector=round_robin#define the sink 1a1.sinks.k1.type=avroa1.sinks.k1.hostname=192.168.11.179a1.sinks.k1.port=9876  #define The sink 2a1.sinks.k2.type=avroa1.sinks.k2.hostname=192.168.11.178a1.sinks.k2.port=9876# use a channel which buffers events in Memorya1.channels.c1.type = memorya1.channels.c1.capacity = 1000a1.channels.c1.transactioncapacity = # Bind The Source and sink to the Channela1.sources.r1.channels = C1a1.sinks.k1.channel = C1a1.sinks.k2.channel=c1

The above is the Flume Agent client configuration, server-side configuration is the same as the previous article http://blog.csdn.net/simonchi/article/details/42494461

Also through the script to simulate the generation of file content, trigger Tail-f, execute two times, found the first output in 179, the second output in 178, indicating that load balancing has taken effect!!

Very simple!!

"Flume" Flume load Balancing Environment construction Load_balance