Introduction to Kafka distributed Message Queue

Similar Products of Kafka distributed Message Queue include JBoss and MQ.

I. It is open-source by javasln and developed using scala. It has the following features:

(1) high throughput

(2) distributed

(3) multi-language clients (C ++ and Java)

Ii. Composition: ClientAre producer and consumer, provide some APIs,ServerYes. The client can publish or consume messages to the broker, and the server can store messages.

Kafka supports partitioning, distribution, and scalability.

3. Kafka messages are divided into several layers

(1) Topics

(2) partition by default, each message has two partitions. You can specify the number of partitions when creating a topic, and divide 0.1 billion rows into eight partitions in one day. If there are hundreds of thousands of rows each day, you can create one partition.

(3) message is a message.

Iv. Data Processing Process

1. The producer produces messages and publishes messages to the specified topic partition.

2. After the Kafka cluster receives the message sent by the producer, it persists the message to the hard disk, which can indicate the scheduled duration, regardless of whether the message is consumed.

3. The consumer uses the pull or push mode of the Kafka cluster and controls the offset of the acquired message. When the consumer restarts, it needs to consume data again based on the offset. The consumer maintains its own offset.

V. How Kafka achieves high throughput

1. Make full use of sequential read/write of disks
2. Batch Data Transmission
3. Data Compression
4. Topics are divided into multiple partitions.

6. How to Implement load balance & ha in Kafka

1) The producer sends messages to the specified partition based on the algorithm specified by the user.
2) Multiple partitions exist, and each partition has multiple replica copies, each of which is distributed on different broker nodes.
3) lead partition must be selected for each partition. Leader partition is responsible for reading and writing, and zookeeper is responsible for fail over fast failure.
4) Manage the dynamic addition and exit of broker and consumer through zookeeper

VII. Resizing

When a broker node needs to be added, the newly added broker will register with zookeeper, and the producer and consumer will perceive these changes based on the watcher on zookeeper and make timely adjustments.

Introduction to Kafka distributed Message Queue