Introduction to Kafka distributed Message Queue

Source: Internet
Author: User

Similar Products of Kafka distributed Message Queue include JBoss and MQ.

I. It is open-source by javasln and developed using scala. It has the following features:

(1) high throughput

(2) distributed

(3) multi-language clients (C ++ and Java)

Ii. Composition: ClientAre producer and consumer, provide some APIs,ServerYes. The client can publish or consume messages to the broker, and the server can store messages.

Kafka supports partitioning, distribution, and scalability.

3. Kafka messages are divided into several layers

(1) Topics

(2) partition by default, each message has two partitions. You can specify the number of partitions when creating a topic, and divide 0.1 billion rows into eight partitions in one day. If there are hundreds of thousands of rows each day, you can create one partition.

(3) message is a message.

Iv. Data Processing Process

1. The producer produces messages and publishes messages to the specified topic partition.

2. After the Kafka cluster receives the message sent by the producer, it persists the message to the hard disk, which can indicate the scheduled duration, regardless of whether the message is consumed.

3. The consumer uses the pull or push mode of the Kafka cluster and controls the offset of the acquired message. When the consumer restarts, it needs to consume data again based on the offset. The consumer maintains its own offset.

V. How Kafka achieves high throughput

1. Make full use of sequential read/write of disks
2. Batch Data Transmission
3. Data Compression
4. Topics are divided into multiple partitions.

6. How to Implement load balance & ha in Kafka

1) The producer sends messages to the specified partition based on the algorithm specified by the user.
2) Multiple partitions exist, and each partition has multiple replica copies, each of which is distributed on different broker nodes.
3) lead partition must be selected for each partition. Leader partition is responsible for reading and writing, and zookeeper is responsible for fail over fast failure.
4) Manage the dynamic addition and exit of broker and consumer through zookeeper

VII. Resizing

When a broker node needs to be added, the newly added broker will register with zookeeper, and the producer and consumer will perceive these changes based on the watcher on zookeeper and make timely adjustments.

Introduction to Kafka distributed Message Queue

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.