Kafka Data Reliability and consistency analysis

1.Partition recovery mechanism

Each partition records a recoverypoint on the disk, recording the maximum offset that has been flush to disk. When the broker fail restarts, Loadlogs is performed. The RecoveryPoint of the partition is read first, the segment that contain recoverypoint are found, and later segment, which is probably not completely flush to disk segment. Then call segment's recover, re-read each segment's MSG, and rebuild the index

Advantages

1. Manage partition data in segment, easy to manage data lifecycle, delete outdated data

2. When the program crashes and restarts, speed up the recovery, simply restore the segment that is not fully flush to the disk

3. Through the index of offset and physical offset mapping, binary lookup can quickly locate the MSG, and by splitting multiple segment, each index file is small, find faster. 2.Partition replica synchronization mechanism

One of the 1.Partition multiple replica is leader and the rest is follower
2.Producer only interacts with leader to write data to leader
3.Followers pull data from leader for data synchronization
4.Consumer pulls data from leader only

ISR: All replica collection, not backward has two meanings: distance from the last fetchrequest time is not greater than a certain value or backward message number is not greater than a certain value, leader failure will choose a follower from the ISR to do leader

About replica replication details: Kafka Copy synchronization mechanism understanding 3. Data Reliability Assurance

When producer sends data to leader, the data reliability Level 1.0 can be set through the ACKs parameter: If the write is successful, the server does not need to send response to producer, and if an exception occurs, the server terminates the connection. Trigger producer update meta data; 2.1:leader Send response If the write succeeds, if leader fail, the data will be lost 3.-1: Wait for all ISR to receive the message and then send producer to response, which is the strongest guarantee only set acks=- 1 There is no guarantee that data is not lost, and when only leader in the ISR list, it is also possible to cause data loss. To ensure that the data is not lost in addition to set acks=-1, but also to ensure that the size of the ISR is greater than or equal to 2, the specific parameter settings: 1.request.required.acks: Set to 1 wait for all the ISR list replica received the message after the write success; 2.min.insync.replicas: set to greater than or equal to 2 to ensure that there are at least two replica producer in the ISR to make a tradeoff between throughput and data reliability 4. Data Consistency Assurance

Consistency definition: If a message is visible to consumer, even if leader is down, the data on the new leader can still be read

1.HighWaterMark short hw:partition High water level, take a partition corresponding ISR in the smallest Leo as HW, consumers can only spend up to the location of the HW, in addition to each replica has Highwatermark, Leader and follower each are responsible for updating their Highwatermark status, highwatermark<= leader. Logendoffset

2. For leader the newly written msg,consumer cannot be consumed immediately, leader will wait for the message to be updated by the replica in all the ISR, then the message can be consumer consumption, that is, consumer can only be consumed to the HW location

This ensures that if the leader broker fails, the message can still be retrieved from the leader of the new election. There is no HW limit for read requests from the internal broker. At the same time, Follower will maintain a copy of his hw,folloer.hw = min (leader.hw, follower.offset)