Flume-kafka-storm-hdfs-hadoop-hbase

# Bigdata-test

Project Address: Https://github.com/windwant/bigdata-test.git

Hadoop:

    Hadoop HDFS Operations

    Log output to Flume

    Flume output to HDFs

HBase

    Htable Basic operations: Create, delete, add table, row, column family, column, etc.

Kafka

    Test Producer | Consumer

Storm: Processing messages in real time


Kafka Integrated Storm Integrated HDFs

    Read Kafka data = "Storm real-time processing (split character, statistical character) =" Write to HDFs


* Kafka Message Generation Method:

*   1. Loggenerator generate test log sent to Flume= "

*   2. Mykafkaproducer Sending test messages

*   3. elasticjob Timer task module generate test log


* MYKAFKASTORMHDFS processing messages in real time:

*   = "Read Kafka data

*   = "storm real-time processing (split characters, statistical characters)

*   = "Write to HDFs

Flume-kafka-storm-hdfs-hadoop-hbase