Flume Sink Processor

Sink groups makes several different sink a whole, while sink processor provides the ability to load balance and failover within a group.

There are three kinds of sink processor:d efault sink processor,failover sink processor,load balancing sink processor.

Default Sink processor

The general sink of the individual

Failover Sink Processor

A sinks priority list is maintained to ensure that only one sink event can be processed (that is, failover).

Sink priority is activated, and priority is determined in the order in which SNK is declared if no priority is set.

Settings are as follows:

Inks	–	Multiple sink are separated by spaces.
Processor.type	default	The name of the component, which must be:failover
Processor.priority.<sinkname>	–	The priority value. <sinkName> must be defined in the sinks. A high priority value of sink will be activated earlier. The greater the value, the higher the priority. Note : If more than one sinks, the priority value should not be the same, if the priority level is the same, only one will take effect. and failover, the same priority will not be failover, even if the same priority exists will also be reported all sinks failed to process.
Processor.maxpenalty	30000	Failed sink maximum backoff time (in milliseconds) (Backoff algorithm (Backoff algorithm) provides a good waiting idea for us when we try to solve a task. ), reference: http://qiuqiang1985.iteye.com/blog/1513049)

Example:

G1K1 K2failover510000    

Load balancing Sink Processor

Provides multiple sinks load balancing capabilities. It maintains an index list of active sinks, and the load of Fenb sinks in the list must be distributed.

Distributed load is implemented by Round_robin (poll) or random (random) selection mechanism . The selection mechanism defaults to Round_robin , and can also be overridden by setting overloads. Custom election classes must inherit Abstractsinkselector.

When called, the selector picks the next sink based on the selection mechanism of the configuration file and calls the sink. If the selected sink Pass event fails, the selection mechanism picks the next available sink, and so on. The failed sink will not be added to the blacklist, and the selector will continue to try all available sink. After all the called Sink fail, the selector sends the failure to sink runner.

If the Backoff property is available, sink processor will darken the failed sink and remove them from the selection within the given timeout period.

Settings are as follows:

Processor.sinks	–	Multiple sink are separated by spaces.
Processor.type	default	The name of the component, which must be:load_balance
Processor.backoff	False	Whether to backoff the failed sinks in the form of an exponent.
Processor.selector	round_robin	Selection mechanism. Must be round_robin , random or a custom class, that inherits theAbstractSinkSelector
Processor.selector.maxTimeOut	30000	The default is 30000 milliseconds, shielding the fault sink time

Example:

G1K1 K2load_balancetruerandom   

Flume Sink Processor