Kafka programming example

Programming

A producer is an application that creates messages and sends them to the Kafka broker. These producer are essentially different. For example, front-end applications, backend services, proxy services, adapters for potential systems, hadoop for producer. These different producers can be implemented in different languages, such as Java, C, and python. The following figure illustrates the Kafka API of the message producer.

The following describes how to compile a simple producer and consumer application.

Send a simple message to the Kafka broker. The producer end writes clusterproducer.

public classClusterProducer extends Thread {    private static final Log log =LogFactory.getLog(ClusterProducer.class);     public void sendData() {        Random rnd = new Random();        Properties props =PropertiesParser.getProperties(PropertiesSettings.PRODUCER_FILE_NAME);        if (props == null) {            log.error("can't loadspecified file " + PropertiesSettings.PRODUCER_FILE_NAME);           return;        }        //set the producer configurationproperties        ProducerConfig config = newProducerConfig(props);        Producer<String, String> producer= new Producer<String, String>(config);         //Send the data        int count = 1;        KeyedMessage<String, String>data;        while (count < 100) {            String sign = "*";            String ip = "192.168.2."+ rnd.nextInt(255);            StringBuffer sb = newStringBuffer();            for (int i = 0; i < count; i++){                sb.append(sign);            }            log.info("set data:" +sb);            try {                Thread.sleep(10);            } catch (InterruptedException e) {                e.printStackTrace();            }            data = new KeyedMessage<String,String>(PropertiesSettings.TOPIC_NAME, ip, sb.toString());            producer.send(data);            count++;        }        producer.close();    }     public void run() {        sendData();    }     public static void main(String[] args) {        new ClusterProducer().sendData();    }}

It is scheduled for the consumer to get the data of the corresponding topic:

public class Consumerextends Thread {    private static final Log log =LogFactory.getLog(Consumer.class);    private final ConsumerConnector consumer;    private final String topic;     public Consumer(String topic) {        consumer =kafka.consumer.Consumer.createJavaConsumerConnector(                createConsumerConfig());        this.topic = topic;    }     private static ConsumerConfigcreateConsumerConfig() {        Properties props = new Properties();       props.put("zookeeper.connect", KafkaProperties.zkConnect);        props.put("group.id",KafkaProperties.groupId);       props.put("zookeeper.session.timeout.ms", "400");       props.put("zookeeper.sync.time.ms", "200");       props.put("auto.commit.interval.ms", "1000");         return new ConsumerConfig(props);     }     public void run() {        Map<String, Integer>topicCountMap = new HashMap<String, Integer>();        topicCountMap.put(topic, newInteger(1));        Map<String,List<KafkaStream<byte[], byte[]>>> consumerMap =consumer.createMessageStreams(topicCountMap);        KafkaStream<byte[], byte[]>stream = consumerMap.get(topic).get(0);        ConsumerIterator<byte[], byte[]>it = stream.iterator();        while (it.hasNext()) {            log.info("+message: " +new String(it.next().message()));        }    }     public static void main(String[] args) {        Consumer client = new Consumer("cluster_statistics_topic");        client.run();    }}

Run the above Code separately to send or obtain the corresponding topic information.

Enjoy yourself! (* ^__ ^ *)......

Kafka programming example