Storm integrated Kafka Data source

Before looking at the contents of this section, we recommend that you look at the first two sections first. Note Install the storm and Kafka version issues .

maven Project Pom.xml add dependencies

<dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <v
      ersion>3.8.1</version> <scope>test</scope> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> &LT;VERSION&G t;1.0.4</version> </dependency> <dependency> &LT;GROUPID&GT;ORG.APACHE.STORM&LT;/GROUPID&G
      T <artifactId>storm-kafka</artifactId> <version>1.0.4</version> </dependency> &L T;dependency> <groupId>org.apache.kafka</groupId> &LT;ARTIFACTID&GT;KAFKA_2.10&LT;/ARTIFACTID&G
      T <version>0.8.2.2</version> <exclusions> <exclusion> <groupid>org.apa
        Che.zookeeper</groupid> <artifactId>zookeeper</artifactId> </exclusion> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> &L T;/exclusion> </exclusions> </dependency>

Kafkatopology.java

Import Org.apache.storm.Config;
Import Org.apache.storm.LocalCluster;
Import Org.apache.storm.StormSubmitter;
Import org.apache.storm.kafka.*;
Import Org.apache.storm.spout.SchemeAsMultiScheme;

Import Org.apache.storm.topology.TopologyBuilder;
Import java.util.ArrayList;
Import java.util.List;

Import Java.util.UUID;
    /** * Kafka 0.8.x.x * */public class Kafkatopology {private static final String kafka_spout_id = "Kafka-stream";

    private static final String parse_bolt_id = "Parse-bolt";

    private static final String Topology_name = "Word-count-topology";
        public static void Main (string[] args) throws Exception {String zkhosts = "zoo1:2181";//zoo1:2181,zoo2:2181
        String topicname= "TOPIC3";
        String zkroot = "/storm"; String Zkspoutid = "Login1";
        The read status will be present,/zkroot/id below, so ID similar to consumer group brokerhosts hosts = new zkhosts (zkhosts);
      Spoutconfig spoutconfig = new Spoutconfig (hosts, Topicname, Zkroot, Zkspoutid);  Spoutconfig.scheme = new Schemeasmultischeme (new Stringscheme ());
        list<string> servers = new arraylist<string> ();
        Servers.add ("Zoo1"); spoutconfig.zkservers = servers;//record spout read progress using the host of zookeeper (must be set, otherwise cannot record progress) Spoutconfig.zkport = 2181;//record progress with Zo

        Okeeper Port (must be set, otherwise the progress cannot be recorded) kafkaspout spout = new Kafkaspout (spoutconfig);
        Parsebolt Parsebolt = new Parsebolt ();
        Topologybuilder builder = new Topologybuilder ();
        Builder.setspout (kafka_spout_id, SPOUT);

        Builder.setbolt (parse_bolt_id, Parsebolt). shufflegrouping (kafka_spout_id);
        Config config = new config ();
        Config.setnumworkers (1);
            if (Args.length = = 0) {localcluster cluster = new Localcluster ();
            Cluster.submittopology (topology_name, config, builder.createtopology ());
            Utils.sleep (100000);
            Cluster.killtopology (Topology_name);
   Cluster.shutdown ();     } else {stormsubmitter.submittopology (args[0], config, builder.createtopology ());
 }
    }
}

Note Spoutconfig.zkservers,spoutconfig.zkport two configurations.
Many tutorials on the Web do not have these two settings, causing the read progress record to fail, each time the program runs from the first data read.

Show Read Data
Parsebolt.java

Import Org.apache.storm.task.OutputCollector;
Import Org.apache.storm.task.TopologyContext;
Import Org.apache.storm.topology.OutputFieldsDeclarer;
Import Org.apache.storm.topology.base.BaseRichBolt;
Import Org.apache.storm.tuple.Tuple;

Import Java.util.Map;

public class Parsebolt extends Baserichbolt {
    private outputcollector collector;

    @Override public
    Void Prepare (Map stormconf, Topologycontext context, Outputcollector collector) {
        This.collector=collector;
    }

    @Override public
    Void execute (Tuple input) {
        String word = input.getstring (0);
        SYSTEM.OUT.PRINTLN (word);
        This.collector.ack (input);//tell kafkaspout that processing is complete (must answer spout to record read progress)
    }

    @Override public
    Void Declareoutputfields (Outputfieldsdeclarer declarer) {

    }
}

It is also important to note that the This.collector.ack (input) answer must be called to tell Kafkaspout that the processing has been completed before kafkaspout will log the read progress, or restart the program and re-read the record.

Executes producer on the server and enters characters at the command line

./bin/kafka-console-producer.sh--broker-list h1:9092--topic topic1

You can see the console panel output data immediately