Kafka Series (ii) features and common commands

Replicas replication backup mechanism in Kafka

Kafka copy each partition data to multiple servers, any one partition has one leader and multiple follower (can not), the number of backups can be set through the broker configuration file ( Replication-factor parameter configuration specified). Leader handles all Read-write requests, follower needs to be synchronized with leader. Follower and consumer, consumer messages are stored in the local log, leader is responsible for tracking all follower status, if follower "lag" too much or fail, Leader will remove it from the Replicas Sync list. When all follower save a message successfully, the message is considered "committed", then consumer can consume it. Even if only one replicas instance survives, The message can still be sent and received normally, as long as the zookeeper cluster survives.

Elections in the Kafka

When the leader fails, it is necessary to select a new leader in the followers, perhaps follower behind the leader, so you need to choose a "up-to-date" follower. When choosing follower, you need to take into account a problem is the number of partition leader that are already hosted on the new leader server, and if there is too much partition leader on one server, it means that the server will be under more IO pressure. In the election of new leader, consideration should be given to "Load balancing".

Common commands

1. Create topics

./kafka-topics.sh--create--zookeeper chenx02:2181--replication-factor 1--partitions 1--topic test

2. View Queue List

./kafka-topics.sh--list--zookeeper chenx02:2181

3. View queue Details

./kafka-topics.sh--describe--zookeeper chenx02:2181--topic test

Results:

Topic:test partitioncount:1 replicationfactor:1 configs:
Topic:test partition:0 leader:2 replicas:2 isr:2

4. Modify (test) queue parameters

./kafka-topics.sh--zookeeper chenx02:2181--partition 3--topic test--alter

Results:

Topic:test partitioncount:3 replicationfactor:1 configs:
Topic:test partition:0 leader:2 replicas:2 isr:2
Topic:test partition:1 leader:1 replicas:1 isr:1
Topic:test Partition:2 leader:2 replicas:2 isr:2

5. Create a queue for multiple replicas

./kafka-topics.sh--create--zookeeper chenx02:2181--replication-factor 3--partitions 4--topic Test_kafka

./kafka-topics.sh--describe--zookeeper chenx02:2181--topic Test_kafka
Topic:test_kafka partitioncount:4 replicationfactor:3 configs:
Topic:test_kafka partition:0 leader:1 replicas:1,3,2 isr:1,3,2
Topic:test_kafka partition:1 leader:2 replicas:2,1,3 isr:2,1,3
Topic:test_kafka Partition:2 leader:3 replicas:3,2,1 isr:3,2,1
Topic:test_kafka Partition:3 leader:1 replicas:1,2,3 isr:1,2,3

Description

Partiton:partion ID
Leader: Current lead broker ID for Read and write
Replicas: All replication Broker list for current partition
A subset of Isr:relicas that contains only the broker for the active state

6. Delete Kafka Queue [note the need to restart Kafka cluster]
kafka-run-class.sh kafka.admin.DeleteTopicCommand--topic Test_kafka--zookeeper chenx02:2181

7. View unavailable partitions
kafka-topics.sh--describe--unavailable-partitions--zookeeper chenx02:2181--topic Test_kafka

8. Send a message
./kafka-console-producer.sh--broker-list chenx02:9092--topic test

9. Consumer News
./kafka-console-consumer.sh--zookeeper chenx02:2181--topic test--from-beginning

Kafka Series (ii) features and common commands