flume1.8 Use Guide Learning sentiment (ii)

1. Flume Sinks1.1 HDFS Sink

The sink writes events into the Hadoop Distributed File System (HDFS). It currently supports the creation of text and sequence files. It supports compression in both file types. Files can be periodically scrolled based on the elapsed time or size of the data or the number of events. It also divides the data into buckets or extents through attributes such as timestamps or machines that occur.

Example of Agent A1:

The above configuration will reduce the timestamp to the last 10 minutes. For example, an event with a 11:54:34 am,june 12,2012 timestamp will cause the HDFs path to become/flume/events/2012-06-12/1150/00.

1.2 Hive Sink

The sink streams transfers the events containing the segmented text or JSON data directly to the Hive table or partition. Write events using hive transactions. When a series of events is submitted to hive, they can be queried by hive immediately.

Hive Table Example:

Agent A1 Example:

The above configuration will reduce the timestamp to the last 10 minutes. For example, the event timestamp header is set to 11:54:34 AM, June 12,2012, and the ' Country ' header is set to ' India ', the event evaluates to partition (continent= ' Asia ', CO Untry= ' India ', time= ' 2012-06-12-11-50 ').

1.3 Logger Sink

Logs event at info level. Typical usage is testing or debugging.

Agent A1 Example:

1.4 Avro Sink

Flume events is sent to sink, converted to Avro events, and sent to the configured hostname/port. Get events in bulk from the configured channel in batches of configured sizes.

Agent A1 Example:

1.5 Thrift Sink

Flume events is sent to sink, converted to thrift events, and sent to the configured hostname/port. Get events in bulk from the configured channel in batches of configured sizes.

Agent A1 Example:

1.5 IRC Sink

IRC sink gets messages and push messages from the linked channel to the configured IRC destination.

Agent A1 Example:

1.6 File Roll Sink

Store events on the local file system.

Agent A1 Example:

1.7 Null Sink

Discard all events when the channel is received.

Agent A1 Example:

1.7 HBaseSinks1.7.1 Hbasesink

The sink writes data to HBase.

Agent A1 Example:

1.7.2 Asynchbasesink

The sink uses asynchronous mode to write data to HBase.

Agent A1 Example:

1.8 Morphlinesolrsink

The sink extracts data from flume events and transforms it in real-time on the Apache SOLR server, and Apache SOLR servers provides query services for end users or search applications.

Agent A1 Example:

1.9 Elasticsearchsink

The sink writes data to the Elasticsearch cluster.

Agent A1 Example:

1.10 Kite Dataset Sink

Test sink write event to Kite Dataset.

1.11 Kafka Sink

The Flume sink implementation can export data to a Kafka topic.

Kafka Sink Configuration Example:

1.11.1 Security and Kafka Sink

The contact channel between Flume and Kafka supports secure authentication and data encryption.

The Security authentication Sasl/gssapi (Kerberos V5) or SSL (TLS implementation) can be used for Kafka version 0.9.0.

Data encryption is provided by SSL/TLS.

Set Kafka.producer.security.protocol to the following value:

• Sasl_plaintext-kerberos or plaintext authentication with no data encryption
• Sasl_ssl-kerberos or plaintext authentication with data encryption
• SSL-TLS based encryption with optional authentication.

TLS and Kafka Sink:

Examples with server-side authentication and data encryption configuration:

Note: The attribute ssl.endpoint.identification.algorithm is not defined, so there is no hostname validation, in order to be hostname authentication, you can set the property:

If client authentication is required, add the following configuration in the Flume agent configuration. Each flume agent must have its client credentials in order to be trusted by Kafka brokers.

If KeyStore and key are protected with a password that is not used, then the Ssl.key.password attribute needs to be provided:

Kerberos and Kafka Sink:

The Kerberos configuration file can be specified in flume-env.sh through java_opts:

Example of a security configuration using Sasl_plaintest:

Example of a security configuration using Sasl_ssl:

Jaas file instance (not yet read):

1.12 HTTP Sink

The sink will get events from the channel and send these events to the remote service using an HTTP POST request. The event content is sent as the post body.

Agent A1 Example:

1.13 Custom Sink (customized Sink)

Custom sink is your implementation of the sink interface. When you start the Flume agent, a custom sink class and its dependencies must be in the agent's classpath.

Agent A1 Example:

Resources:

Https://flume.apache.org/FlumeUserGuide.html

flume1.8 Use Guide Learning sentiment (ii)