Spring Integrated Kafka

Main references Spring open source project: Spring for Apache Kafka

This article mainly introduces the use of consumer and peoducer, other such as connector, stream, etc., can check the official website documents, development should pay attention to the primary key version compatibility, here my spring version is

<spring.version>4.3. . Release</spring.version>

First, add the Kafka related jar to the Pom file:

 for kafka--><dependency>    <groupId>org.springframework.kafka</groupId>    <artifactId>spring-kafka</artifactId>    <version>2.1.  6. Release</version></dependency>

Next you'll show how to consume the messages in Kafka:

First write the Consumer listener class Kafkaconsumerlisten.java (callback method without consumer instance):

Package Com.yinz.kafka;import Org.apache.kafka.clients.consumer.consumerrecord;import Org.springframework.kafka.listener.MessageListener;  Public class messagelistener<string, string>{    @Override    publicvoid OnMessage (consumerrecord<string, string> data) {        System. out. println ("--------------" + Data.value ());}    }

Kafkaconcurconsumerlisten.java (callback method with consumer instance):

Package Com.yinz.kafka;import Org.apache.kafka.clients.consumer.consumer;import Org.apache.kafka.clients.consumer.consumerrecord;import Org.springframework.kafka.listener.ConsumerAwareMessageListener; Public classKafkaconcurconsumerlisten ImplementsConsumerawaremessagelistener<string, string>{@Override Public voidOnMessage (consumerrecord<string, string> data, consumer<?,? >consumer) {System. out. println (Consumer.hashcode () +">>>>>>"+ data.partition () +">>>>>>>>>>"+Data.value ()); }}

Next configure the related bean, here I Kafka the relevant configuration is in Kafka.xml:

The contents of the Kafka.xml file are as follows ( both individual consumers and concurrent consumers are configured in real-time use ):

<?xml version="1.0"encoding="UTF-8"? ><beans xmlns="Http://www.springframework.org/schema/beans"Xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"Xmlns:context="Http://www.springframework.org/schema/context"xsi:schemalocation="Http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd http//Www.springframework.org/schema/contexthttp://www.springframework.org/schema/context/spring-context-4.3.xsd"><!--define the producer class--<!--define producer Parameters--<bean id="producerproperties" class="Java.util.HashMap"> <constructor-arg> <map> <entry key="bootstrap.servers"Value="47.98.131.199:9092"/> <entry key="Retries"Value="1"/> <!--message send failed, retry count--<entry key="batch.size"Value="16384"/> <!--message package send, size cannot exceed this value-<entry key="linger.ms"Value="1"/> <!--the millisecond is delayed before the message is sent, and when the load is heavier it can reduce the number of times the request is sent, that is, the package sends multiple messages at once--<entry key="buffer.memory"Value="33554432"/> <!--producer buffer size, used to store messages that have not yet been sent to the broker, that the value will wait, and then be thrown one time later--<entry key="Key.serializer"Value="Org.apache.kafka.common.serialization.StringSerializer"/> <entry key="Value.serializer"Value="Org.apache.kafka.common.serialization.StringSerializer"/> </map> </constructor-arg> </bean> <!--create kafkatemplate producer you need to use Factory Bean--<bean id="producerfactory" class="org.springframework.kafka.core.DefaultKafkaProducerFactory"> <constructor-argref="producerproperties"></constructor-arg> </bean> <bean id="kafkatemplate" class="org.springframework.kafka.core.KafkaTemplate"> <constructor-argref="producerfactory"/> <constructor-arg name="AutoFlush"Value="false"/> <!--trueSignificantly reduces performance-<property name="Defaulttopic"Value="Test"/> <!--default topic--</bean> <!--define consumer parameters--<bean id="consumerproperties" class="Java.util.HashMap"> <constructor-arg> <map> <entry key="bootstrap.servers"Value="47.98.131.199:9092"/> <entry key="group.id"Value="0"/> <!--consumer group ID--<entry key="Enable.auto.commit"Value="false"/> <!--do not automatically submit offset-to-<entry key="auto.commit.interval.ms"Value=" +"/> <!--regularly submit offset interval-<entry key="session.timeout.ms"Value="15000"/> <!--15 seconds without a heartbeat means consumer--<entry key="Key.deserializer"value="Org.apache.kafka.common.serialization.StringDeserializer"/> <entry key="Value.deserializer"value="Org.apache.kafka.common.serialization.StringDeserializer"/> </map> </constructor-arg> </bean> <!--create Consumerfactory Bean--&gt    ; <bean id="consumerfactory"        class="org.springframework.kafka.core.DefaultKafkaConsumerFactory"> <constructor-arg> <refbean="consumerproperties"/> </constructor-arg> </bean> <!--classes that actually execute message consumption, there are no consumer instances in the callback method--<!--
<bean id="Messagelisternerconsumerservice" class="Com.yinz.kafka.KafkaConsumerListen"/> <!--Consumer container configuration information--<bean id="containerproperties" class="org.springframework.kafka.listener.config.ContainerProperties"> <constructor-arg value="Test"/> <property name="MessageListener" ref="Messagelisternerconsumerservice"/> </bean> <!--Configure individual consumers in the same group, only one consumer, unable to achieve load averaging and disaster recovery -<bean id="Messagelistenercontainer" class="Org.springframework.kafka.listener. Kafkamessagelistenercontainer "Init-method="Dostart"> <constructor-argref="consumerfactory"/> <constructor-argref="containerproperties"/> </bean>
-<!--the class that actually executes the message consumption, there are consumer instances in the callback method--<bean id="Messageconcurlisternerconsumerservice" class="Com.yinz.kafka.KafkaConcurConsumerListen"/> <bean id="containerconcurproperties" class="org.springframework.kafka.listener.config.ContainerProperties"> <constructor-arg value="Test-par"/> <property name="MessageListener" ref="Messageconcurlisternerconsumerservice"/> </bean> <!--Configure concurrency consumers in the same group,can achieve load averaging and disaster recovery, if concurrency is greater than the top partition number, it will be automatically degraded-<bean id="messageConcurlistenercontainer" class="Org.springframework.kafka.listener. Concurrentmessagelistenercontainer "Init-method="Dostart"> <constructor-argref="consumerfactory"/> <constructor-argref="containerConcurProperties"/> <property name="Concurrency"Value="3"></property> </bean></beans>

The producer is relatively simple, just inject kafkatemplate Bean, call his send method to send a message.

Spring Integrated Kafka