Logger = Loggerfactory.getlogger ( This. GetClass ()); @KafkaListener (Topics= {"Test"}) Public voidListen (consumerrecordrecord) {Logger.info ("Kafka key:" +Record.key ()); Logger.info ("Kafka Value:" +Record.value (). toString ()); }}Tips1) I did not describe how to install the configuration Kafka, the best way to configure
1. Start the production and consumption process using 127.0.0.1:
1) Start the producer process:
bin/kafka-console-producer.sh--broker-list 127.0.0.1:9092--topic test
Input message:
This is MSG
Producer Process Error:
[2016-06-03 11:33:47,934] WARN Bootstrap broker 127.0.0.1:9092 Disconnected (org.apache.kafka.clients.NetworkClient)
[2016-06-03 11:33:49,554] WARN Bootstrap broker 127.0.0.1:9092 Disconnected (org.apache.kafka.clients.NetworkClient)
For Kafkaconsumer, it is not like kafkaproducer, not thread-safe, the state is maintained in the consumer, so the implementation should pay attention to the use of multi-threading, generally there are 2 ways to use: 1: Each consumer has its own thread, Consumer to pull data, and processing, this method is relatively simple, easy to implement, easy to process mess
Original:https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+ExampleWhy use the high level Consumer
In some scenarios, we want to read messages through multithreading, and we don't care about the order in which messages are consumed from Kafka, we only care about the data being consumed. High leve
If you are using Kafka to distribute messages, there may be exceptions or other errors in the process of data processing that can result in loss or inconsistency. This time you may want to Kafka the data through the new process, we know that Kafka by default will be saved on disk to 7 days of data, you just need to Kafka
Original:https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+ExampleWhy use the high level Consumer
In some scenarios, we want to read messages through multithreading, and we don't care about the order in which messages are consumed from Kafka, we just care about the data being consumed. High leve
At present the Central Library Org.apache.kafka is compiled with jdk1.7, so run in 1.6 of the JVM will be errorSolution:1. Download Kafka source code, local SBT to install, pre-compile java-version confirm that the JDK version in Classpath is 1.62. After compiling the package successfully, go to the core/target/scala_2.8.0/of the current Kafka directory, find
consumer crashes trigger rebalancing, and when the new consumer tries to update the offset information, it will cause the missing message, as shown in:Pull and consume of messagesConsumer if a message exists during the last message pull, it is returned directly, otherwise the pull cancellation request is resent from the previously updated pull offset location.Pull cancellation request to the
I here Kafka consumer code is copied online, is to open a thread monitoring Kafka topic, a message on the processing. The code to begin with is this:
public void Kafkastart () {final String topic = HipchatAction.properties.getProperty ("Kafka.hipchat.topic"); Final int partitionnum = integer.valueof (HipchatAction.properties.getProperty ("Kafka.hipchat.topic.par
Reprint please specify original address http://www.cnblogs.com/dongxiao-yang/p/6238029.htmlRecently, we need to study in detail the algorithm details of the partition calculation in the process of Kafka Reblance, searching some of the words on the internet, feeling more obscure and not easy to understand, or self-keying the source code more convenient.The Kafka Reblance calculates part of the code as follow
); Mapconsumerconnector.createmessagestreams (map); List); //Create List of 4 threads to consume from each of the partitionsExecutorservice executor = Executors.newfixedthreadpool (4); //consume the messages in the threads for(FinalKafkastreamstream:streams) {Executor.submit (NewRunnable () { Public voidrun () { for(Messageandmetadata msgandmetadata:stream) {//Process Message (Msgandmetadata.message ())System.out.println ("topic:" +msgandmetadata.topic ()); Message Message=(Message) msgandmeta
CONSUMERFETCHERMANAGER5. Generate timed tasks, according to the configuration of auto.commit.interval.ms to submit the displacement, the default is 1 minutes to submit the displacement to the zookeeper is to periodically save the consumed message displacement to zookeeper, the specific logic is very simple, This article does not dwell on here. We only care about the fourth step in the previous step-creating Consumerfetchermanager. So, what does Consumerfetchermanager do with it? As the name imp
In the use of Kafka high-level consumer, the use of multi-threaded consumption data times wrong, simple analysis of the reason for download , Consumeriterator will not be blocked when the message and set the internal state to failed, which throws an exception when other threads access it.Java code 650) this.width=650; "class=" star "src=" Http://woodding2008.iteye.com/images/icon_star.png "alt=" Favorites
The producers are as follows:
public class KafkaProduce1 {public
static void Main (string[] args) throws ioexception{
String topic= "test";
Properties prop = new properties ();
Prop.load (KafkaProduce1.class.getClassLoader (). getResourceAsStream ("producer.properties"));
producer
The producer.properies configuration is as follows:
# List of brokers used for bootstrapping knowledge about the rest of the cluster
# FORMAT:HOST1:PORT1,HOST2:PORT2. .
metad
Error process:650) this.width=650; "Src=" https://s1.51cto.com/wyfs02/M00/91/60/wKiom1j1sYHyYBNNAADoGsBpPEc665.png-wh_500x0-wm_ 3-wmp_4-s_1657427620.png "title=" Qq20170418142553.png "alt=" Wkiom1j1syhyybnnaadogsbppec665.png-wh_50 "/>Cause: Host name mapped IP error, resulting in no communication between nodesWorkaround: Configure IP and hostname mappings on each machineVi/etc/hosts650) this.width=650; "Src=" https://s2.51cto.com/wyfs02/M01/91/5F/wKioL1j1skjRcJXNAAAZuUAeA9Q879.png-wh_500x0-wm_ 3
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.