1 Why use Message Queuing?
Six words: Decoupling, asynchronous, peak extinction.
2 What are the disadvantages of using Message Queuing?
Message Queuing is suspended, the system is unusable, and system availability is reduced.
3 high availability of Message Queuing?
Kafka use Zookeeper,master/slave to ensure high availability;
Kafka manages cluster configuration through zookeeper, elects leader, and rebalance when consumer group changes
4 How do I ensure that messages are not repeated consumption ( idempotent )?
Kafka has the concept of offset, that is, every message has a offset,kafka consume the message, you need to submit offset, let the message queue know that they have been consumed;
Cause of repeated consumption: because of network transmission and so on, the confirmation information is not transmitted to the message queue , causing the message queue not to know that they have consumed the message, and again to distribute the message to other consumers. How to avoid:
- Insert operation, the primary key uniqueness can be guaranteed.
- Update, there will be no problem.
- In other cases, you can use a third-party tool to make a consumption record (such as Redis) and assign a global ID to the message. Writes <id,message> to Redis in k-v form. Before consumption, go to Redis to inquire if there are any consumption records.
5 How to ensure the reliability of the transmission of consumption?
In the process of using Message Queuing, we should not consume more information or consume less .
In fact, this reliability transmission, each MQ has to be analyzed from three angles: the producer loses the data , the message queue loses the data, the consumer loses the data
(1) producers lose data:
Producer when publishing a message to a partition, the leader of the partition is first found through zookeeper, and then no matter how much topic replication the factor is (that is, the partition has Number of replica), producer only sends the message to the partition leader. Leader writes the message to its local log. Each follower pull data from the leader
In Kafka production, there is basically a leader and a plurality of follwer. Follwer will synchronize the leader information.
Therefore, to avoid data loss by producers, do the following two-point configuration:
1. The first configuration is to set the Acks=all on the producer side. This configuration ensures that the message is sent successfully after the Follwer synchronization is complete.
2. Set Retries=max on the producer side, once the write fails, this infinite retry;
(2) Message queue data loss:
For the message queue to lose data, no outside of IS, the data is not synchronized, leader hung up, then Zookpeer will be the other follwer switch to leader, the data is lost. For this scenario, you should do two configurations.
1. Replication.factor parameter, this value must be greater than 1, which requires that each partition must have at least 2 copies
2. Min.insync.replicas parameter, this value must be greater than 1, this is to require a leader at least to perceive that there is at least one follower still keep in touch with himself
These two configurations, combined with the above producer's configuration, basically ensure that the Kafka does not lose data.
(3) The consumer loses the data: the consumer loses the data generally is because uses the automatic confirmation message pattern, as to the solution, uses the manual confirmation message to be able.
6 How do I guarantee the order of messages?
for this problem, by some algorithm, you need to keep sequencing messages in the same message queue (Kafka is the queue in PARTITION,RABBITMQ). Then use only one consumer to consume the queue (in Kafka, a partition can only be consumed by a consumer in a consumer group).
Six Message Queue Review