Kafka distributed Deployment and verification

Kafka is a messaging component in a distributed environment, and Kafka message components cannot be used if Kafka application processes are killed or Kafka machines are down.

Kafka Cluster (cluster)

A machine is not enough, then more than a few, first of all, start zookeeper this will not say more. You can refer to the previous article, when starting the Kafka, we in a stand-alone simulation to start multiple Kafka applications.

First in the config directory, copy two server.properties files, here I copy three copies, named Server1.properties, Server2.properties server3.properties

Then modify these three configuration files, mainly modify the Broker.id=2,port=9094,log.dir=/tmp/kafka-logs-2 three values, broker.id is used to mark the broker in the distributed environment, requiring unique, Ports and Log.dir a port, a log directory, if it is not required to be modified in a real-world distributed environment. This is a stand-alone simulation to prevent port collisions.

Change Broker.id to 1,2,3,log.dir, respectively, to kafka-logs-1,kafka-logs-2,kafka-logs-3, then start

Kafka-server-start.bat.. /.. /config/server1.properties

Kafka-server-start.bat.. /.. /config/server2.properties

Kafka-server-start.bat.. /.. /config/server3.properties

If you start to have an error, one is that the VM parameter is too large, another may be your port has not been modified. It's good to see the error.

And then we sign up for a topic, called Replicationtest.

Kafka-topics.bat--create--zookeeper localhost:2181--replication-factor 3--partitions 1--topic replicationtest

Here the redundancy is 3, the partition is 1, then the end of each broker will be retained, the maximum allowable N-1, that is, 2 broker downtime, the service is still running.

After registering, this time

Kafka-topics.bat--describe--zookeeper localhost:2181--topic replicationtest

Execute the description command and look at the cluster situation

Cluster description Results

The first line gives the summary information for the partition. Each partition line gives the partition information.

The "Leader" node is 2.

"Replicas" information, on the node 2,3,1, all node information.

A collection of replication nodes in the "ISR" work. A collection of live nodes.

The rest is not to be explained. 2 is leader, which means that 2 of this node will provide services to consumers.

Then we test a piece of information.

Kafka-console-producer.bat--broker-list localhost:7777,localhost:8888,localhost:9999--topic replicationtest

The 7777 above is set in server1.properties. According to personal circumstances, change. Then send a message at the console.

And then spend it.

Kafka-console-consumer.bat--zookeeper localhost:2181--topic replicationtest

Here's 2181 is the zookeeper port, do not have to change.

Current Run results

And then. We started to turn off a broker and Ctrl,c in the 3 console. Then whether to terminate the operation, enter Y.

and send a message.

Broker3 Downtime

Everything's fine. Let's take a look at cluster information

Broker3 Downtime Cluster

3 fewer machines were found in the ISR because 3 were dead.

Then we turn off the broker2. At this point, a new leader election will be triggered. The expectation 1 becomes leader, sends another message

Broker2 Downtime

You can see that the producer sent a message with an exception because the connection to 2 was disconnected. But note that the message was not lost because the new election was triggered. As you can see, consumers are still receiving normal news. The cluster is as follows

Broker2 Downtime Cluster

At this point, Kafka's broker cluster test is over, and the rest of the problem comes. When the consumer starts up, it connects to the zookeeper address, if the zookeeper hangs.

Then we need to zookeeper cluster deployment.

Zookeeper Cluster

This includes two parts.

Is that the broker would have to know the addresses of these zookeeper clusters and switch to another zookeeper when a downtime occurs.

Consumers need to know the address of these zookeeper, for the same reason.

So the steps are as follows. You can try it yourself, it's easier.

Copy 3 copies of the Zookeeper.properties file, named Zookeeper1.properties,zookeeper2.properties,zookeeper3.properties, and modify the dataDir= in the file /tmp/zookeeper and, clientport=2181, ports are set to 2181,2182,2183 respectively. Then start three zookeeper

Modify the Kafka boot configuration, server1.properties the zookeeper.connect=localhost:2181 this configuration in three files, separated by commas. Eventually Zookeeper.connect=localhost : 2181,localhost:2182,localhost:2183, then start

The producer also changes the. metadata.broker.list=localhost:9092 in the configuration, and if you start with the command line, you don't have to change it. Parameter designation is also possible.

Consumers can change the zookeeper.connect=127.0.0.1:2181 in the configuration file and change the command line when it starts.

5. The end result is a variety of downtime tests.