Multiplexing technology for "Flume" Flume multiplexing

Source: Internet
Author: User

Multiplexing technology is intended to send an event to a specific channel based on configuration information.

A source instance can specify multiple channels, but a sink instance can only specify one channel.

Flume supports fanning out the flow from one source to multiple channels. There is modes of fan out, replicating and multiplexing

Flume supports two modes of output from source to multiple channel: copy and Reuse

1. In copy mode, the event data received by source is output to all channel configured

2. In multiplexing mode, the event data received by source is output to a specific channel configured

The first choice of this mode, flume through the selector selector, the default is the copy mode

# list The sources, sinks and channels in the agentagent_foo.sources = Avro-appsrv-source1agent_foo.sinks = hdfs-cluster1- Sink1 avro-forward-sink2agent_foo.channels = mem-channel-1 file-channel-2# set channels for Sourceagent_ Foo.sources.avro-appsrv-source1.channels = mem-channel-1 file-channel-2# set channel for Sinksagent_ Foo.sinks.hdfs-cluster1-sink1.channel = Mem-channel-1agent_foo.sinks.avro-forward-sink2.channel = file-channel-2# Channel Selector configurationagent_foo.sources.avro-appsrv-source1.selector.type = Multiplexingagent_ Foo.sources.avro-appsrv-source1.selector.header = stateagent_foo.sources.avro-appsrv-source1.selector.mapping.ca = Mem-channel-1agent_foo.sources.avro-appsrv-source1.selector.mapping.az = File-channel-2agent_ Foo.sources.avro-appsrv-source1.selector.mapping.ny = Mem-channel-1 File-channel-2agent_ Foo.sources.avro-appsrv-source1.selector.default = Mem-channel-1
Think of this is a multiplexed channel mode, where the header is the header of the event data, according to the value of the header to determine the output to which channel.

Of course, the above scenario applies to the case where the data source collection code is coupled to its own application code, because you cannot determine the contents of the header data for this event until the data is collected by other means without the code involved in the decoupling.


Copy mode is much simpler, slightly!


Multiplexing technology for "Flume" Flume multiplexing

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.