Sink groups makes several different sink a whole, while sink processor provides the ability to load balance and failover within a group.
There are three kinds of sink processor:d efault sink processor,failover sink processor,load balancing sink processor.
Default Sink processor
The general sink of the individual
Failover Sink Processor
A sinks priority list is maintained to ensure that only one sink event can be processed (that is, failover).
Sink priority is activated, and priority is determined in the order in which SNK is declared if no priority is set.
Settings are as follows:
Inks |
– |
Multiple sink are separated by spaces. |
Processor.type |
default |
The name of the component, which must be:failover |
Processor.priority.<sinkname> |
– |
The priority value. <sinkName> must be defined in the sinks. A high priority value of sink will be activated earlier. The greater the value, the higher the priority. Note : If more than one sinks, the priority value should not be the same, if the priority level is the same, only one will take effect. and failover, the same priority will not be failover, even if the same priority exists will also be reported all sinks failed to process. |
Processor.maxpenalty |
30000 |
Failed sink maximum backoff time (in milliseconds) (Backoff algorithm (Backoff algorithm) provides a good waiting idea for us when we try to solve a task. ), reference: http://qiuqiang1985.iteye.com/blog/1513049) |
Example:
G1K1 K2failover510000
Load balancing Sink Processor
Provides multiple sinks load balancing capabilities. It maintains an index list of active sinks, and the load of Fenb sinks in the list must be distributed.
Distributed load is implemented by Round_robin (poll) or random (random) selection mechanism . The selection mechanism defaults to Round_robin , and can also be overridden by setting overloads. Custom election classes must inherit Abstractsinkselector.
When called, the selector picks the next sink based on the selection mechanism of the configuration file and calls the sink. If the selected sink Pass event fails, the selection mechanism picks the next available sink, and so on. The failed sink will not be added to the blacklist, and the selector will continue to try all available sink. After all the called Sink fail, the selector sends the failure to sink runner.
If the Backoff property is available, sink processor will darken the failed sink and remove them from the selection within the given timeout period.
Settings are as follows:
Processor.sinks |
– |
Multiple sink are separated by spaces. |
Processor.type |
default |
The name of the component, which must be:load_balance |
Processor.backoff |
False |
Whether to backoff the failed sinks in the form of an exponent. |
Processor.selector |
round_robin |
Selection mechanism. Must be round_robin , random or a custom class, that inherits theAbstractSinkSelector |
Processor.selector.maxTimeOut |
30000 |
The default is 30000 milliseconds, shielding the fault sink time |
Example:
G1K1 K2load_balancetruerandom
Flume Sink Processor