Kafka message middleware and Java example

Kafka is a message middleware for passing messages between systems, and messages can be persisted!

Can be considered as a queue model, but also can be seen as a producer consumption model;

The simple producer consumer client code is as follows:

 PackageCom.pt.util.kafka;Importjava.util.Date;Importjava.util.Properties;ImportKafka.javaapi.producer.Producer;ImportKafka.producer.KeyedMessage;ImportKafka.producer.ProducerConfig; Public classMyproducer { Public Static voidsendmsg (String msg) {Properties props=NewProperties (); //Brokers ListProps.put ("Metadata.broker.list", "192.168.91.231:9092,192.168.91.231:9093"); /** The serializer when preparing the message for transmission to the Broker * Note that the encoder          Must accept the same type * as defined in the Keyedmessage object in the next step. *          */Props.put ("Serializer.class", "Kafka.serializer.StringEncoder"); /** * Defines what class-to- determine * which Partition in the Topic the message was to be sent to*/Props.put ("Partitioner.class", "Example.producer.SimplePartitioner"); /** * Tells Kafka that's want your Producer to require a * acknowledgement from the Broker that T He message was received*/Props.put ("Request.required.acks", "1"); Producerconfig Config=Newproducerconfig (props); /** Note that the Producer are a Java Generic and you need to tell it the type of the parameters.         * The first is the type of the Partition key, the second the type of the message. */Producer<string, string> producer =NewProducer<string, string>(config); Longruntime =NewDate (). GetTime (); String IP= "192.168.91.231"; /** the "Panteng" is the Topic to write to.         * Here we are passing the IP as the partition key. * Note that if you don't include a key, * even if you ' ve defined a Partitioner class, Kafka'll assign the mess         Age to a random partition. */Keyedmessage<string, string> data =NewKeyedmessage<string, string>(                "Panteng", IP, msg);        Producer.send (data);    Producer.close (); }}

Producer,java

 PackageCn.outofmemory.kafka;ImportJava.util.HashMap;Importjava.util.List;ImportJava.util.Map;Importjava.util.Properties;ImportKafka.consumer.ConsumerConfig;ImportKafka.consumer.ConsumerIterator;ImportKafka.consumer.KafkaStream;ImportKafka.javaapi.consumer.ConsumerConnector;ImportKafka.serializer.StringDecoder;Importkafka.utils.VerifiableProperties; Public classKafkaconsumer {Private FinalConsumerconnector Consumer;  PublicKafkaconsumer () {Properties props=NewProperties (); //Zookeeper ConfigurationProps.put ("Zookeeper.connect", "192.168.91.231:2181"); //Group represents a consumer groupProps.put ("Group.id", "Jd-group"); //ZK Connection timed outProps.put ("zookeeper.session.timeout.ms", "4000"); Props.put ("Zookeeper.sync.time.ms", "200"); Props.put ("Auto.commit.interval.ms", "1000"); Props.put ("Auto.offset.reset", "smallest"); //Serialization ClassesProps.put ("Serializer.class", "Kafka.serializer.StringEncoder"); Consumerconfig Config=Newconsumerconfig (props); Consumer=kafka.consumer.Consumer.createJavaConsumerConnector (config); }     Public voidconsume () {Map<string, integer> topiccountmap =NewHashmap<string, integer>(); Topiccountmap.put ("Panteng",NewInteger (1)); Stringdecoder Keydecoder=NewStringdecoder (Newverifiableproperties ()); Stringdecoder Valuedecoder=NewStringdecoder (Newverifiableproperties ()); Map<string, list<kafkastream<string, string>>> consumermap =Consumer.createmessagestreams (Topiccountmap,keydecoder,valuedecoder); Kafkastream<string, string> stream = Consumermap.get ("Panteng"). Get (0); Consumeriterator<string, string> it =Stream.iterator ();  while(It.hasnext ()) System.out.println (It.next (). message ()); }         Public voidStop () {Try{consumer.shutdown (); } Catch(Exception e) {//Todo:handle ExceptionE.printstacktrace (); }            }     Public Static voidMain (string[] args) {NewKafkaconsumer (). consume (); }}

Consumer.java

Kafka message middleware and Java example