Flume Learning---Data flow model__flume

A/Flume data flow model

Flume event is defined as a data flow unit with byte payload and optional string properties, and the Flume agent is the JVM process that hosts the components of an event from the external source to the next destination. The following figure is the flume agent flowchart

Flume Source consumes events that are passed to him by an external source, such as a Web Server.
The external source is sent to the Flume in a formatted event and can be identified by the target Flume source.
such as: Avro Source: A stream similar to the Arvo event event that is sent from a Avro client or other Avro sink can be defined using thrift Flume Source to receive from thrift sink or thrift RPC clients , or thrift clients, which are written in any language based on the thrift Protocol.

When Flume Source receives the event, it saves it to one or more channels.
Flume Channel is a passive storage. Save the event until it is consumed by flume sink,
If the file Channel:file channels is supported by the local filesystem, Sink removes the event from the channel, writes it to the external store, or forwards the source in the next flume agent .

The source and sink within the given agent run asynchronously with the events staged in the channel.

Complex Data Flow

Flume allows a user to build Multi-hop flows where events travel through multiple agents before reaching the final Destina tion. It also allows Fan-in and fan-out flows, contextual Routing and Backup routes (Fail-over) for failed hops.

ii. Reliability (reliability)

The event is stored in the channel of each agent. The event is then passed to the next agent or final repository (such as HDFs) in the stream. Event will not be removed from channel until it is stored in the channel of the next agent or in the final repository. This is how the single hop message passing semantics in Flume provides end-to-end reliability of the stream.
Flume uses the transaction method (transactional approach) to ensure reliable delivery of the event. Source and sink encapsulate, respectively, storage/retrieval of event that is placed or provided by a channel-provided transaction in a transaction. This ensures that the event collection is reliably delivered from point to point in the stream. In the case of multi-hop flow, the sink from the previous hop and source from the next hop run their transactions to ensure that the data is securely stored in the next-hop channel. iii. recoverability (recoverability)

Event storage load channel, management recovers from failure. Flume supports persistent file channel supported by the local file system. There is also a memory channel that simply stores the event in a memory queue, which is quicker, but any event that remains in the memory channel when the agent process dies is not recoverable.