Introduction to the new features of the Apache Kafka 0.11 release

Apache Kafka recently launched version 0.11. This is a major milestone version, especially Kafka from this release to support the "Exactly-once" semantics (hereinafter called EOS, Exactly-once semantics). This article briefly describes the major functional changes in the 0.11 release, each of which deserves a special article for a good chat.

First, modify the unclean.leader.election.enabled default value

The Kafka community is finally determined to change the default value of this parameter to False, that is, the unclean leader election is no longer allowed, and the former is chosen between correctness and high availability. If you still want to enable it, the user needs to explicitly set this parameter in Server.properties =true

Second, ensure that the Offsets.topic.replication.factor parameters are applied correctly

__consumer_offsets This topic is created automatically by Kafka when created, if the cluster broker number < Offsets.topic.replication.factor, the original version takes its small, but this will violate the user's intention to set the parameter. As a result, this parameter will be enforced in version 0.11, and group_coordinator_not_available will be thrown if the value set by this parameter is not met.

Third, optimize the support of snappy compression

Before because the source code hard-coded block size, so that producer use snappy when the performance is much different than LZ4, but in fact snappy and LZ4 the gap should not be very large. Therefore, the default block size for snappy is adjusted in version 0.11. This, however, requires a detailed performance test report to prove that the change is valid.

Iv. Message Increase header information (header)

The record adds a header, and each header is a KV store. Specific header design refer to KIP-82

Five, empty consumer group delay rebalance

In order to shorten the time for the first rebalance of multiple consumer, the "group.initial.rebalance.delay.ms" is used to set the delay time for group open rebalance. This delay period allows more consumer to join the group, avoiding unnecessary switching between joingroup and Syncgroup. Of course everything is trade-off, introduce this inevitably bring consumption delay.

Vi. Change of message format

Add the latest Magic value: 2. Added header information. In order to support idempotent producer and EOS, additional transaction-related fields are added, resulting in an increase in the volume of a single record data structure. However, due to the optimized recordbatch, the total volume of batch is reduced, further reducing the network IO overhead.

Vii. New allocation algorithm: Stickyassignor

A more balanced allocation algorithm than range and Round-robin. Specify Partition.assignment.strategy = Org.apache.kafka.clients.consumer.StickyAssignor to taste fresh. However, according to my experience, uneven distribution usually occurs when each consumer subscription topic is very different. such as Consumer1 subscription Topic1, Topic2, TOPIC4, Consumer2 subscription topic3, topic4 this situation

Eight, controller re-design

The controller's original design was complex, making it almost impossible for people in the community to change the controller code.  The main problem with older controllers seems to me to have 2 of them: 1. Controller needs to perform 1,2,3,4,5,6 step operation, if the 3rd step error, cannot rollback the first two steps of operation; 2. Multi-threaded access, with multiple threads accessing the controller context information at the same time. Version 0.11 partially reconstructs the controller, using a single-threaded + event queue-based approach. The concrete effect we wait and see ~ ~

Nine, support EOS

0.11 most important features, no one! EOS is the cornerstone for the correctness of streaming. The mainstream streaming framework basically supports EOS (such as Storm Trident, Spark streaming, Flink), and Kafka streams must also support it. Version 0.11 supports eos:1. Producer with 3 major changes (this is also the function of awaited); 2. Support Services; 3. EOS-enabled streaming (guaranteed read-process-write EOS for full-link)

Introduction to the new features of the Apache Kafka 0.11 release