Flume Sink Processor

Source: Internet
Author: User
Tags failover

Sink groups makes several different sink a whole, while sink processor provides the ability to load balance and failover within a group.

There are three kinds of sink processor:d efault sink processor,failover sink processor,load balancing sink processor.

Default Sink processor

The general sink of the individual

Failover Sink Processor

A sinks priority list is maintained to ensure that only one sink event can be processed (that is, failover).

Sink priority is activated, and priority is determined in the order in which SNK is declared if no priority is set.

Settings are as follows:

Inks Multiple sink are separated by spaces.
Processor.type default The name of the component, which must be:failover
Processor.priority.<sinkname> The priority value. <sinkName> must be defined in the sinks. A high priority value of sink will be activated earlier. The greater the value, the higher the priority.
Note : If more than one sinks, the priority value should not be the same, if the priority level is the same, only one will take effect. and failover, the same priority will not be failover, even if the same priority exists will also be reported all sinks failed to process.
Processor.maxpenalty 30000 Failed sink maximum backoff time (in milliseconds) (Backoff algorithm (Backoff algorithm) provides a good waiting idea for us when we try to solve a task. ), reference: http://qiuqiang1985.iteye.com/blog/1513049)

Example:

G1K1 K2failover510000    
Load balancing Sink Processor

Provides multiple sinks load balancing capabilities. It maintains an index list of active sinks, and the load of Fenb sinks in the list must be distributed.

Distributed load is implemented by Round_robin (poll) or random (random) selection mechanism . The selection mechanism defaults to Round_robin , and can also be overridden by setting overloads. Custom election classes must inherit Abstractsinkselector.

When called, the selector picks the next sink based on the selection mechanism of the configuration file and calls the sink. If the selected sink Pass event fails, the selection mechanism picks the next available sink, and so on. The failed sink will not be added to the blacklist, and the selector will continue to try all available sink. After all the called Sink fail, the selector sends the failure to sink runner.

If the Backoff property is available, sink processor will darken the failed sink and remove them from the selection within the given timeout period.

Settings are as follows:

Processor.sinks Multiple sink are separated by spaces.
Processor.type default The name of the component, which must be:load_balance
Processor.backoff False Whether to backoff the failed sinks in the form of an exponent.
Processor.selector round_robin Selection mechanism. Must be round_robin , random or a custom class, that inherits theAbstractSinkSelector
Processor.selector.maxTimeOut 30000

The default is 30000 milliseconds, shielding the fault sink time

Example:

G1K1 K2load_balancetruerandom   

Flume Sink Processor

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.