Six Message Queue Review

Source: Internet
Author: User
Tags zookeeper

1 Why use Message Queuing?

  Six words: Decoupling, asynchronous, peak extinction.

2 What are the disadvantages of using Message Queuing?

  Message Queuing is suspended, the system is unusable, and system availability is reduced.

3 high availability of Message Queuing?

  Kafka use Zookeeper,master/slave to ensure high availability;

Kafka manages cluster configuration through zookeeper, elects leader, and rebalance when consumer group changes

4 How do I ensure that messages are not repeated consumption ( idempotent )?

Kafka has the concept of offset, that is, every message has a offset,kafka consume the message, you need to submit offset, let the message queue know that they have been consumed;

Cause of repeated consumption: because of network transmission and so on, the confirmation information is not transmitted to the message queue , causing the message queue not to know that they have consumed the message, and again to distribute the message to other consumers. How to avoid:

    • Insert operation, the primary key uniqueness can be guaranteed.
    • Update, there will be no problem.
    • In other cases, you can use a third-party tool to make a consumption record (such as Redis) and assign a global ID to the message. Writes <id,message> to Redis in k-v form. Before consumption, go to Redis to inquire if there are any consumption records.

5 How to ensure the reliability of the transmission of consumption?

  In the process of using Message Queuing, we should not consume more information or consume less .

In fact, this reliability transmission, each MQ has to be analyzed from three angles: the producer loses the data , the message queue loses the data, the consumer loses the data

(1) producers lose data:

    Producer when publishing a message to a partition, the leader of the partition is first found through zookeeper, and then no matter how much topic replication the factor is (that is, the partition has Number of replica), producer only sends the message to the partition leader. Leader writes the message to its local log. Each follower pull data from the leader

In Kafka production, there is basically a leader and a plurality of follwer. Follwer will synchronize the leader information.

Therefore, to avoid data loss by producers, do the following two-point configuration:

      1. The first configuration is to set the Acks=all on the producer side. This configuration ensures that the message is sent successfully after the Follwer synchronization is complete.

2. Set Retries=max on the producer side, once the write fails, this infinite retry;

(2) Message queue data loss:

For the message queue to lose data, no outside of IS, the data is not synchronized, leader hung up, then Zookpeer will be the other follwer switch to leader, the data is lost. For this scenario, you should do two configurations.

1. Replication.factor parameter, this value must be greater than 1, which requires that each partition must have at least 2 copies

2. Min.insync.replicas parameter, this value must be greater than 1, this is to require a leader at least to perceive that there is at least one follower still keep in touch with himself

These two configurations, combined with the above producer's configuration, basically ensure that the Kafka does not lose data.

(3) The consumer loses the data: the consumer loses the data generally is because uses the automatic confirmation message pattern, as to the solution, uses the manual confirmation message to be able.

6 How do I guarantee the order of messages?

for this problem, by some algorithm, you need to keep sequencing messages in the same message queue (Kafka is the queue in PARTITION,RABBITMQ). Then use only one consumer to consume the queue (in Kafka, a partition can only be consumed by a consumer in a consumer group).

     

  

Six Message Queue Review

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.