kafka java example

java.util.map;import Java.util.properties;import Java.util.concurrent.executorservice;import Java.util.concurrent.executors;import Kafka.consumer.consumer;import Kafka.consumer.consumerconfig;import Kafka.consumer.consumeriterator;import Kafka.consumer.kafkastream;import Kafka.javaapi.consumer.consumerconnector;import Kafka.message.messageandmetadata;public class Logconsumer {private Consumerconfig config; Private String topic; private int partitionsnum; Private Messageexecutor exec

(true) {String message = "message-" + ++count;//消息主题是testKeyedMessagenew KeyedMessage"test", message);//message可以带key, 根据key来将消息分配到指定区, 如果没有key则随机分配到某个区// KeyedMessageproducer.send(keyedMessage);System.out.println("send: " + message);try {Thread.sleep(1000);} catch (InterruptedException e) {e.printStackTrace();}}// producer.close();} }/*** 自定义分区类**/class MyPartition implements Partitioner {public int partition(Object key, int numPartitions) {return key.hashCode()%numPartitions;}} From fo

;ImportKafka.javaapi.consumer.ConsumerConnector;ImportKafka.serializer.StringDecoder;Importkafka.utils.VerifiableProperties; Public classKafkaconsumer {Private FinalConsumerconnector Consumer; PublicKafkaconsumer () {Properties props=NewProperties (); //Zookeeper ConfigurationProps.put ("Zookeeper.connect", "192.168.91.231:2181"); //Group represents a consumer groupProps.put ("Group.id", "Jd-group"); //ZK Connection timed outProps.put ("zookeeper.session.timeout.ms", "4000"); Props.put ("Zookee

Recently want to test the performance of Kafka, toss a lot of genius to Kafka installed to the window. The entire process of installation is provided below, which is absolutely usable and complete, while providing complete Kafka Java client code to communicate with Kafka. He

number of message streams to return. number of streams to be returned * @ param keyDecoder a decoder that decodes the message key can be decoded Key decoder * @ param valueDecoder a decoder that decodes the message itself can decode the decoder of the message itself * @ return a list of KafkaStream. each stream supports an * iterator over its MessageAndMetadata elements. returns the KafkaStream list. Each stream supports an iterator Based on the MessagesAndMetadata element. */ public /*** Cre

Flume and Kakfa example (KAKFA as Flume sink output to Kafka topic)To prepare the work:$sudo mkdir-p/flume/web_spooldir$sudo chmod a+w-r/flumeTo edit a flume configuration file:$ cat/home/tester/flafka/spooldir_kafka.conf# Name The components in this agentAgent1.sources = WeblogsrcAgent1.sinks = Kafka-sinkAgent1.channels = Memchannel# Configure The sourceAgent1.s

[TOC] Kafka Note Finishing (ii): Kafka Java API usageThe following test code uses the following topic:$ kafka-topics.sh --describe hadoop --zookeeper uplooking01:2181,uplooking02:2181,uplooking03:2181Topic:hadoop PartitionCount:3 ReplicationFactor:3 Configs: Topic: hadoop Partition: 0 Le

, consuming input streams from 1 or more topic, and producing an output stream to 1 or more topic, effectively swapping the input flow to the output stream. The connector API allows you to build or run reusable producers or consumers and link topic to existing applications or data systems. The example diagram is as follows:Kafka Application Scenarios Build real-time streaming data pipelines that can reliably get data between systems

Kafka 0.9 version of the Java Client API made a large adjustment, this article mainly summarizes the Kafka 0.9 in the cluster construction, high availability, the new API related processes and details, as well as I in the installation and commissioning process to step out of the various pits.About Kafka structure, func

In the previous section (Point this transfer), we completed the Kafka cluster, in this section we will introduce the new API in version 0.9, and the test of Kafka cluster high availability1. Use Kafka's producer API to complete the push of messages1) Kafka 0.9.0.1 Java Client dependency:2) Write a Kafkautil tool class

The environment of this article is as follows:Operating system: CentOS 6 32-bitJDK version: 1.8.0_77 32-bitKafka version: 0.9.0.1 (Scala 2.11) 1. Maven Dependency Packagedependency> groupId>org.apache.kafkagroupId> artifactId>kafka-clientsartifactId> version>0.9.0.1version>dependency>2. Producer CodePackagecom. Lnho. Example. Kafka;import org. Apach

) Conf directory, then copy zoo_sample.cfg to zoo.cfg4) Modify the Datadir=d:\zookeeper-3.3.6\zookeeper-3.3.6\data in Zoo.cfg(according to the decompression path to adjust accordingly)3. Start Zookeeper go to the bin directory and execute Zkserver.cmdOpen command Window in Bin directory: SHIFT + right mouse buttonInput: Zkserver.cmd Enter execution4 Kafka ConfigurationCompressed Package Decompression: D:\kafka_2.11-0.11.0.1Go to config directory, edit

First, download zookeeper and Kafkafrom the official website(the locally used version is zookeeper-3.3.6, kafka_2.11-1.0.0):Second, configure zookeeper and Kafka and start, basic zkcli command and Kafka Create Delete topic Command. 2.1 Configuration zookeeper, the main configuration has two, one is the port 2181, the other is the data storage path. 2.2 start zookeeper2.3 Use the zookeeper client to view T

In the Kafka download page, download version 0.8, unzip. 1. Modify the server.properties inside the Config directory to host.name the IP of the machine. If the deployment Kafka and the development run Kafka example for the same machine without modification, with the default localhost also line. 2. Modify the DataDir pr

Kafka Consumer API Example 1. Auto-confirm OffsetDescription Reference: http://blog.csdn.net/xianzhen376/article/details/51167333Properties Props = new properties ();/* Defines the address of the KAKFA service and does not require all brokers to be specified on */props. put ("Bootstrap.servers","localhost:9092");/* Develop consumer group */props. put ("Group.id","Test");/* Whether to automatically confirm t

Decoration in ...KafkaNew TopicBin/kafka-topics. sh --create--zookeeper localhost:218131 --topic my-topicView a list of existing topicBin/kafka-topics. sh --list--zookeeper localhost:2181View the specified topic statusBin/kafka-topics. sh --describe--zookeeper localhost:2181 --topic my-topicStart consumer read message and output to standard outputBin/

/ui.log2>1/data0/xxx/storm-0.9.5/bin/storm logviewer >/data0/xxx/storm-0.9.5/log/logviewer.log2>1 Start Nimbus, supervisor, UI, LogviewerExample Storm configuration file: Storm.zookeeper.servers: -"172.16.0.100" nimbus.host: "172.16.0.100" storm.local.dir: "/data0/xxx/ Storm-0.9.5/workdir " storm.messaging.netty.max_retries:30 storm.messaging.netty.min_wait_ms:100 storm.messaging.netty.max_wait_ms:1500 supervisor.slots.ports: -6700 -6701 -6702 -6703 supervisor.worker.start.timeout.se

("message"). ToString (). Contains ("A")) println ("Find A in message:" +map.tostring ())}}classRulefilelistenerbextendsStreaminglistener {override Def onbatchstarted (batchstarted: org.apache.spark.streaming.scheduler.StreamingListenerBatchStarted) {println ("-------------------------------------------------------------------------------------------------------------- -------------------------------") println ("Check whether the file's modified date is change, if change then reload the configu

same category are sent to the same topic and then consumed by topic consumer. Topic is a logical concept, and the physical realization is partition. (3) Partition: partition. Partitioning is a physical concept, each topic contains one or more partition, and each partition is an ordered queue . Messages sent to topic are partitioned (customizable) to determine which partition the message is stored in. Each piece of data will be assigned an ordered id:offset. Note:

Java provides a convenient API for Kafka message processing. Briefly summarize:Study reference:http://www.itnose.net/st/6095038.htmlPOM Configuration (see http://www.cnblogs.com/huayu0815/p/5341712.html for log4j configuration)　　PRODUCERImport Kafka.javaapi.producer.producer;import Kafka.producer.keyedmessage;import kafka.producer.ProducerConfig; Import Java.util.properties;public class Kafkaproducer {pr