Spring Cloud Building MicroServices Architecture (VII) Message bus (cont.: Kafka)

In addition to supporting RABBITMQ's automated configuration, Spring Cloud bus supports Kafka, which is now widely used. In this article, we will build a Kafka local environment and use it to try the support of Spring Cloud Bus for Kafka to realize the function of message bus. Since this article will be modified based on the implementation of the previous rabbit, it is helpful to read the "Spring Cloud Building MicroServices Architecture (VII) message bus" To help understand this article.

Kafka Introduction

Kafka is a distributed messaging system developed by LinkedIn, open source in early 2011 and now maintained and developed by the prestigious Apache Foundation. Kafka is implemented in Scala and is used as a conduit for LinkedIn's activity streams and operational data processing, and is now widely used by many internet companies as a data flow pipeline and messaging system.

Kafka is a message system based on the implementation of the message publish/subscribe pattern, and its main design objectives are as follows:

• Message persistence: Provides message persistence in the form of Time complexity O (1), which guarantees constant-time complexity of access performance even for terabytes or more data.
• High throughput: Supports up to 100K throughput per second on inexpensive commercial machines
• Distributed: Supports message partitioning and distributed consumption, and guarantees the order of messages within a partition
• Cross-platform: Clients that support different technology platforms (e.g. Java, PHP, Python, etc.)
• Real-time: Supports real-time data processing and offline data processing
• Scalability: Supports horizontal scaling

Some of the basic concepts involved in Kafka:

• The Broker:kafka cluster contains one or more servers, which are referred to as broker.
• Topic: Logically similar to rabbit queue queues, each message published to the Kafka cluster must have a Topic. (Physically different topic messages are stored separately, logically a topic message is saved on one or more brokers, but the user only needs to specify the topic of the message to produce or consume data without worrying about where the data is stored)
• Partition:partition is a physical concept of partitioning, in order to provide system throughput, each topic is physically divided into one or more Partition, each Partition corresponding to a folder (store the message content and index file of the corresponding partition).
• Producer: Message producer, responsible for producing messages and sending them to Kafka Broker.
• Consumer: The message consumer, which reads the message to Kafka broker and processes the client.
• Consumer Group: Each Consumer belongs to a specific group (which can be assigned to a group for each Consumer, if not specified, it belongs to the default group), and the group can be used to implement a message that is consumed by multiple members within the group.

Quick Start

After having some basic understanding of Kafka, let's try to build a Kafka server and experience the production and consumption of Kafka-based messages.

Environment installation

First, we need to download the installation media from the official website. For: http://kafka.apache.org/downloads.html . The version used in this example is: Kafka-0.10.0.1

After extracting the Kafka installation package, you can see the following directory structure:

1234567

Kafka+-bin+-windows+-config+-libs+-logs+-site-docs

Since Kafka's design relies on zookeeper, we can bin see zookeeper related content in and out of the config catalog, in addition to seeing Kafka-related content. The bin directory contains the command-line tools for Kafka and zookeeper, and the bin root directory is the shell for Linux/unix, while the bin/windows next one is for bat under Windows. We can set the environment variables according to the actual system to facilitate the subsequent use and operation. In the config directory, it is used to store configuration information about Kafka and zookeeper.

Start test

Let's try to start zookeeper and Kafka to produce and consume messages. All of the commands in the example have been configured with the Kafka environment variable as an example.

• start zookeeper, execute command: zookeeper-server-start config/zookeeper.properties the command needs to specify the Zookeeper profile location to start correctly, the Kafka package contains its default configuration, and the development and test environment does not need to be modified.

123456789

[2016-09-28 08:05:34,849] INFO Reading configuration from:config\zookeeper.properties (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [ 2016-09-28 08:05:34,850] INFO Autopurge.snapretaincount set to 3 (Org.apache.zookeeper.server.DatadirCleanupManager) [ 2016-09-28 08:05:34,851] INFO Autopurge.purgeinterval set to 0 (Org.apache.zookeeper.server.DatadirCleanupManager) [ 2016-09-28 08:05:34,851] INFO Purge task is not scheduled. (Org.apache.zookeeper.server.DatadirCleanupManager) [2016-09-28 08:05:34,852] WARN either no config or no quorum defined in config, running in standalone mode (Org.apache.zookeeper.server.quorum.Quoru Mpeermain) [2016-09-28 08:05:34,868] INFO Reading configuration from:config\zookeeper.properties ( Org.apache.zookeeper.server.quorum.QuorumPeerConfig) [2016-09-28 08:05:34,869] INFO starting server ( Org.apache.zookeeper.server.ZooKeeperServerMain) ... [2016-09-28 08:05:34,940] INFO binding to Port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

From the console information, we can see that zookeeper reads the information from the specified config/zookeeper.properties configuration file and binds the 2181 port to start the service. Sometimes startup fails, you can see if the port is occupied, kill the consuming process, or config/zookeeper.properties start zookeeper by modifying the contents of the configuration file clientPort to bind a different port number.

• start Kafka, execute command: kafka-server-start config/server.properties the command also needs to specify the correct location of the Kafka configuration file, as pointed to in the command above to understand the default configuration contained by the compression directory. If we use the zookeeper of the external centralized environment in the test, we can set the address and port of zookeeper by the parameter in this configuration file zookeeper.connect , it will connect the local 2181 port zookeeper by default, if you need to set up multiple zookeeper nodes , you can configure multiple zookeeper addresses for this parameter, separated by commas. For example: zookeeper.connect=127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002 .

• Create topic, execute command: kafka-topics --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test through this command, create a topic named "Test", which contains a replica for a partition. After the creation is complete, you can use kafka-topics --list --zookeeper localhost:2181 the command to view the current topic.

In addition, if we do not use the kafka-topics command to create manually, the following content directly to create the message is also automatically created topics to use.

• Create the message producer and execute the command: kafka-console-producer --broker-list localhost:9092 --topic test . kafka-console-producercommand to start the Kafka command-line-based message production client, which can be sent directly in the console after it is started, and each row of data in the console is treated as a message. We can try to enter a few lines of messages, because there is no consumer at this time, so the input messages will be blocked in the topics named Test, until a consumer will be consuming their location.

• Create message Consumer , execute command: kafka-console-consumer --zookeeper localhost:2181 --topic test --from-beginning . The kafka-console-consumer command starts with the Kafka command-line-based message consumption client, which we can see in the console immediately after the message that we sent in the message production client. We can again open the previous message production client to send the message, and observe the consumer side of the output of the message to experience the Kafka of the message base processing.

Integrate Spring Cloud Bus

In the previous case of implementing the message bus using rabbit, we have spring-cloud-starter-bus-amqp completed the message bus using RABBITMQ by introducing the module. If we want to use Kafka to implement the message bus, we just need to spring-cloud-starter-bus-amqp replace the spring-cloud-starter-bus-kafka module, in pom.xml the Dependenies node to modify, as follows:

1234	<dependency> <groupid>org.springframework.cloud</groupid> <artifactid>spring-cloud-starter-bus-kafka</artifactid> </dependency>

If we use the default configuration when we start Kafka, we do not need to do any additional configuration to locally implement the switchover from RABBITMQ to Kafka. We can try to start up the zookeeper, Kafka, and the spring-cloud-starter-bus-kafka Config-server and config-client of the module that we just built.

When Config-server is started, we can see the following output in the console:

12345678910111213

2016-09-28 22:11:29.627 INFO 15144---[main] o.s.c.s.b.k.kafkamessagechannelbinder:using Kafka topic for OUTBOUND:SPR ingcloudbus2016-09-28 22:11:29.642 INFO 15144---[-localhost:2181] org. I0Itec.zkclient.ZkEventThread:Starting zkclient Event Thread .... 016-09-28 22:11:30.290 INFO 15144---[main] o.s.i.kafka.support.producerfactorybean:using producer properties = {b ootstrap.servers=localhost:9092, Linger.ms=0, Acks=1, Compression.type=none, batch.size=16384}2016-09-28 22:11:30.298 INFO 15144---[main] o.a.k.clients.producer.producerconfig:producerconfig values: ... 2016-09-28 22:11:30.322 INFO 15144---[main] o.s.c.s.b.k.kafkamessagechannelbinder$1:adding {message-handler:o Utbound.springcloudbus} as a subscriber to the ' springcloudbusoutput ' channel2016-09-28 22:11:30.322 INFO 15144---[main ] O.s.integration.channel.directchannel:channel ' config-server:7001.springcloudbusoutput ' has 1 subscriber (s). 2016-09-28 22:11:30.322 INFO 15144---[main] O.s.c.s.b.k.kafkaMessagechannelbinder$1:started Outbound.springcloudbus ... 2016-09-28 22:11:31.465 INFO 15144---[main] s.i.k.i.kafkamessagedrivenchanneladapter:started org.springframe[email& nbsp;protected]4178cb342016-09-28 22:11:31.467 INFO 15144---[main] o.s.c.s.b.k.kafkamessagechannelbinder$7:adding { MESSAGE-HANDLER:INBOUND.SPRINGCLOUDBUS.ANONYMOUS.8B9E6C7B-6A50-48C5-B981-8282A0D5A30B} as a subscriber to the ' Bridge.springcloudbus ' channel2016-09-28 22:11:31.467 INFO 15144---[main] o.s.c.s.b.k.kafkamessagechannelbinder$7: Started inbound.springcloudbus.anonymous.8b9e6c7b-6a50-48c5-b981-8282a0d5a30b

From the console output, we can see that the config-server is connected to the Kafka and uses the springCloudBus topic named.

At this point, we can use the kafka-topics --list --zookeeper localhost:2181 command to view the current Kafka in the topic, if we have successfully started the Config-server and configured correctly, we can see in the Kafka that there is a more than one named springCloudBus topic.

We then launch the config-client with the spring-cloud-starter-bus-kafka module configured, and you can see the output from the console as follows:

12345678910111213

2016-09-28 22:43:55.067 INFO 6136---[main] o.s.c.s.b.k.kafkamessagechannelbinder:using Kafka topic for OUTBOUND:SPRI ngcloudbus2016-09-28 22:43:55.078 INFO 6136---[-localhost:2181] org. I0Itec.zkclient.ZkEventThread:Starting zkclient Event Thread .... 2016-09-28 22:50:38.584 INFO 828---[main] o.s.i.kafka.support.producerfactorybean:using producer properties = {Bo otstrap.servers=localhost:9092, Linger.ms=0, Acks=1, Compression.type=none, batch.size=16384}2016-09-28 22:50:38.592 INFO 828---[main] o.a.k.clients.producer.producerconfig:producerconfig values: ... 2016-09-28 22:50:38.615 INFO 828---[main] o.s.c.s.b.k.kafkamessagechannelbinder$1:adding {message-handler:o Utbound.springcloudbus} as a subscriber to the ' springcloudbusoutput ' channel2016-09-28 22:50:38.616 INFO 828---[main] O.s.integration.channel.directchannel:channel ' Didispace:7002.springcloudbusoutput ' has 1 subscriber (s). 2016-09-28 22:50:38.616 INFO 828---[main] O.s.c.s.b.k.kafkamessagechannelbInder$1:started Outbound.springcloudbus ... 2016-09-28 22:50:39.162 INFO 828---[main] s.i.k.i.kafkamessagedrivenchanneladapter:started org.springframe[email& nbsp;protected]60cf855e2016-09-28 22:50:39.162 INFO 828---[main] o.s.c.s.b.k.kafkamessagechannelbinder$7:adding { message-handler:inbound.springcloudbus.anonymous.f8fc9c0c-ccd3-46dd-9537-07198f4ee216} as a subscriber to the ' Bridge.springcloudbus ' channel2016-09-28 22:50:39.163 INFO 828---[main] o.s.c.s.b.k.kafkamessagechannelbinder$7: Started inbound.springcloudbus.anonymous.f8fc9c0c-ccd3-46dd-9537-07198f4ee216

As you can see, config-client output similar content at startup, and they all subscribe to the springCloudBus topic named.

After Config-server and Config-client are started, we can start the config-client of many different ports locally in order to see more clearly the effect of the message bus refresh configuration. At this point, our config-server and multiple config-client have been connected to the message bus implemented by Kafka. We can first access the requests on each config-client to /from see what configuration he gets. Then, modify the corresponding parameter content in Git, and then access the requests on each config-client /from , and you can see that the configuration content has not changed. Finally, we send a POST request to Config-server: /bus/refresh at this point we go to the request on each config-client to /from get the latest configuration information, the configuration on each client has been loaded as the latest git configuration content.

From the Config-client console, we can see the following:

1	2016-09-29 08:20:34.361 INFO 21256---[kafka-binder-1] o.s.cloud.bus.event.refreshlistener:received remote Refresh req Uest. Keys refreshed [from]

RefreshListenerThe Listener class records the log that received the remote refresh request and refreshed the from properties.

Kafka Configuration

In the example above, since Kafka and zookeeper are all running locally, we did not experiment with the local message bus by specifying configuration information for Kafka and zookeeper in the test program. But in our actual application, Kafka and zookeeper are generally deployed independently, so they need to configure some connection information for Kafka and zookeeper in the application. Unlike the RABBITMQ, which is not directly available in the spring Boot 1.3.7, the Kafka integrates with the Kafka module of Spring Cloud Stream, so the Kafka configuration uses the The prefix for the Spring.cloud.stream.kafka , such as:

Property name	Description	Default Value
Spring.cloud.stream.kafka.binder.brokers	Server-side list of Kafka	localhost
Spring.cloud.stream.kafka.binder.defaultBrokerPort	The default port on the Kafka server, which is used when the brokers port information is not configured in the property	9092
Spring.cloud.stream.kafka.binder.zkNodes	Kafka server-side Connection Zookeeper node list	localhost
Spring.cloud.stream.kafka.binder.defaultZkPort	The default port for the Zookeeper node, which is used when the zkNodes port information is not configured in the property	2181

For more configuration parameters, please refer to the official documentation

The complete example of this article:

• Open source China: http://git.oschina.net/didispace/SpringCloud-Learning/tree/master/Chapter1-1-7
• Github:https://github.com/dyc87112/springcloud-learning/tree/master/chapter1-1-7

Spring Cloud Building MicroServices Architecture (VII) MESSAGE bus (continued: Kafka)