Using Kafka in Spring Boot

Kafka is a highly huff and puff distributed subscription message system, which can replace the traditional message queue for decoupled data processing, cache unhandled messages, and has higher throughput, support partition, multiple replicas and redundancy, so it is widely used in large-scale message data processing applications. Kafka supports Java and a variety of other language clients and can be used in conjunction with other large data tools such as Hadoop, Storm, and Spark. Environment Installation

Build high throughput Kafka distributed subscription message cluster test case Github code

Code I have put into Github, import Spring-boot-kafka Project

GitHub Https://github.com/souyunku/spring-boot-examples/tree/master/spring-boot-kafka Add dependencies

Add kafka-clients dependencies to your project

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients< /artifactid>
    <version>0.10.2.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactid>spring-kafka</ Artifactid>
</dependency>

Enable Kafka

@Configuration
@EnableKafka Public
class Kafkaconfiguration {

}

message producer

@Component public class Msgproducer {private static final Logger log = Loggerfactory.getlogger (Msgproducer.class);

    @Autowired private kafkatemplate<string, string> kafkatemplate;
        public void SendMessage (string topicname, String jsondata) {log.info ("Push Data to Kafka: [{}]", jsondata);
        try {kafkatemplate.send (topicname, jsondata); catch (Exception e) {log.error ("Error sending data ...)
            {} {} ", topicname, Jsondata);
        Log.error ("Send data Error =====>", e);
            }//message-sent listener for callback return information Kafkatemplate.setproducerlistener (new producerlistener<string, string> () { @Override public void onsuccess (string topic, Integer partition, string key, String value, Recordmet Adata recordmetadata) {} @Override public void OnError (String topic, Integer partiti On, string key, String value, Exception Exception) {} @Override Public Boolean isinterestedinsuccess () {log.info ("Data sent");
            return false;
    }
        }); }

}

Message Consumers

@Component public
class Msgconsumer {

    @KafkaListener (topics = {"Topic-1", "topic-2"}) public
    void ProcessMessage (String content) {

        System.out.println ("message consumed" +content);
    }
    

parameter Configuration

application.properties

#kafka
# Specifies Kafka proxy address, can be multiple
spring.kafka.bootstrap-servers=yz-ptest-app-hadoop-02:9092, yz-ptest-app-hadoop-04:9092
# Specifies the number of threads in the listener container, which is used to increase the amount of concurrency
spring.kafka.listener.concurrency=3
# Number of messages per bulk send
spring.kafka.producer.batch-size=1000
# Specifies default consumer group ID
spring.kafka.consumer.group-id= MyGroup
# Specifies the default topic ID
spring.kafka.template.default-topic=topic-1

Start a service

@SpringBootApplication
@ComponentScan (value = {"Io.ymq.kafka"}) public
class Startup {public

    static void Main (string[] args) {
        springapplication.run (startup.class, args);
    }
}

Unit Test

import Io.ymq.kafka.MsgProducer;
Import Io.ymq.kafka.run.Startup;
Import Org.junit.Test;
Import Org.junit.runner.RunWith;
Import org.springframework.beans.factory.annotation.Autowired;
Import Org.springframework.boot.test.context.SpringBootTest;
Import Org.springframework.test.context.junit4.SpringRunner; /** * Description: Test Kafka * * @author Yanpenglei * @create 2017-10-16 18:45 **/@RunWith (springrunner.class) @SpringBootTest

    (classes = startup.class) public class Basetest {@Autowired private msgproducer msgproducer;
        @Test public void Test () throws Exception {msgproducer.sendmessage ("topic-1", "topic--------1");
    Msgproducer.sendmessage ("Topic-2", "topic--------2"); }
}

Message producer, responding

2017-10-17 15:54:44.814  INFO 2960---[           main] io.ymq.kafka.MsgProducer                 : Push Data to Kafka: [Topic--------1]
2017-10-17 15:54:44.860  INFO 2960---[           main] io.ymq.kafka.MsgProducer                 : Push Data to Kafka: [Topic--------2]
2017-10-17 15:54:44.878  INFO 2960---[ad | producer-1] io.ymq.kafka.MsgProducer                 : Data sent
2017-10-17 15:54:44.878  Info 2960- --[AD | producer-1] io.ymq.kafka.MsgProducer                 : Data sent complete

Message consumers, responding

Messages are consumed topic--------1
messages are consumed topic--------2

Code I have put into Github, import Spring-boot-kafka Project

GitHub Https://github.com/souyunku/spring-boot-examples/tree/master/spring-boot-kafka met some pits

[2017-10-16 19:20:08.340]-14884 critical [main]---org.springframework.kafka.support.LoggingProducerListener:Exception Thrown when sending a-key= ' null ' and payload= ' topic--------2 ' to topic Topic-2:

The host name of the Kafka connection is found by debugging, so modify the hosts

c:\windows\system32\drivers\etc\hosts 10.32.32.149 yz-ptest-app-hadoop-02 10.32.32.154 yz-ptest-app-hadoop-04