Flume study 05

Channel Selector

The channel has two selectors, replicating and multiplexing, and if you do not specify a selector, the replicating selector is used by default. Replicating Selector

The replicating selector copies the event multiple copies, and each channel receives the same event. If source is associated with 3 channel, then the 3 channel will receive the same number of event. Source uses the replicating selector by default.

a1.sources = R1
a1.channels = C1 C2 c3
A1.source.r1.selector.type = replicating
a1.source.r1.channels = C1 C2 C3
A1.source.r1.selector.optional = C3

Multiplexing Selector

The multiplexing is a shunt selector that filters event data and distributes the event to different channel based on the message header properties. Multiplexing is not the default selector and needs to be configured. The following configuration is diverted according to the header property State of the event, if the state is CZ sent to channel C1, if state is us then sent to channel C2 and channel C3, if neither is sent to channel C4.

a1.sources = R1
a1.channels = C1 c2 c3 c4
a1.sources.r1.selector.type = multiplexing
A1.sources.r1.selector.header = state
A1.sources.r1.selector.mapping.CZ = C1
A1.sources.r1.selector.mapping.US = C2 C3
A1.sources.r1.selector.default = C4

Sink Processors

Sink support grouping, can put multiple sink into a group inside, as a whole external services. The sink group has three types of processors, default, failover, and load_balance, respectively.
The default processor supports only a single sink, and if there is only one sink in the packet, the processor for this grouping can be set to default. It is generally not necessary to create a grouping for a single sink.
The failover processor maintains a prioritized list sink with multiple sink in the list, ensuring that event data is processed in a timely manner. The mechanism of failover is that each sink has a continuous failure counter, and when the sink successive failures reach the threshold, the sink is lowered to priority and placed in the cooling pool, sink in the cooling pool for a period of time before being enabled. If sink successfully sends an event, it will be restored to the active pool. Each sink has a priority, the higher the priority, the greater the value, such as 100 priority above 80 priority. If a sink fails to send an event, the sink with the highest priority will attempt to send the failed event.

a1.sinkgroups = G1
a1.sinkgroups.g1.sinks = K1 K2
a1.sinkgroups.g1.processor.type = Failover
A1.SINKGROUPS.G1.PROCESSOR.PRIORITY.K1 = 5
a1.sinkgroups.g1.processor.priority.k2 = ten
A1.sinkgroups.g1.processor.maxpenalty = 10000

The above configuration group has K1, K2 two sink, the processor type is FAILOVER,K1 priority is 5,K2 priority is 10,maxpenalty is the maximum cooldown time.
The load balancing policy has round_robin and random, and the load balancer configuration is as follows.

a1.sinkgroups = G1
a1.sinkgroups.g1.sinks = K1 K2
a1.sinkgroups.g1.processor.type = load_balance
A1.sinkgroups.g1.processor.backoff = true
a1.sinkgroups.g1.processor.selector = random

Flume Interceptors

The flume supports the interceptor chain, which can modify and discard event data (a filter that functions like a Java Web or a spring MVC interceptor). Flume custom interceptors as long as the Org.apache.flume.interceptor.Interceptor interface is implemented. The order of the Interceptor calls is configured in the configuration file, with each interceptor returning the event data list after the call is complete, and if you want to discard the event data, simply remove the data from the list.

The

name	Description
Timestamp Interceptor	Timestamp Interceptor Inserts the current timestamp in the header of the event, the timestamp key is Timestamp, and the value is the current time.
host Intercerptor	host Interceptor Inserts the host name or IP address of the current agent in the header of the event. The key value of the header is host by default, or it can be modified in configuration item Hostheader.
static Interceptor	static interceptors allow you to add some fixed content to the head, each static interceptor can only add one header content, if you want to add multiple header content, you need to have multiple static interceptors. The
uuid Interceptor	UUID Interceptor adds a unique sequence to the header of each event. The header key is the ID by default and can be modified in the configuration item headername.
morphline Interceptor
Search and Replace interceptor	The Interceptor finds and replaces the text content in the event data with regular expressions.
Regex Filtering Interceptor	This interceptor matches the text content in the event with a regular expression, enabling filtering of the event.
Regex Extractor Interceptor	The interceptor matches the text content in the event with a regular expression and writes the matched content to the header.