Apache Kafka Source Analysis-Autoleaderrebalanceenable

In the broker's configuration,auto.leader.rebalance.enable (false)

So how does this leader rebalance?

First, a scheduler is opened when the controller is started,

if // If you open the outoleaderrebalance, you need to Partiton leader because of the dead and migration, re-migration back        Info ("Starting the partition rebalance Scheduler")        Autorebalancescheduler.startup ()        Autorebalancescheduler.schedule ("Partition-rebalance-thread", Checkandtriggerpartitionrebalance,          5, Config.leaderimbalancecheckintervalseconds, timeunit.seconds)      }

To do it regularly,

Checkandtriggerpartitionrebalance

This function logic is to find out all the replica that have been migrated, namely

Topicsnotinpreferredreplica

and determine if the imbalance ratio is met, that is, automatically trigger leader rebalance, will leader moved back to Perfer replica

The key to understand what is preferred replicas?

Preferredreplicasfortopicsbybrokers =          = = = Deletetopicmanager.istopicqueuedupfordeletion (p._1. Topic). groupBy            {case (topicandpartition, Assignedreplicas) = assignedreplicas.head          }

Topicandpartition can uniquely identify a partition,seq[int] by topic name and partition ID, which means brokerids, Which shows the replicas of the partition on which brokers

Filter out the delete topic from partition's Replicaassignment, and then follow Assignedreplicas.head GroupBy, which is the first brokerid in SEQ
This means that the default preferred replica for each partition is the first assign replica

The result of GroupBy is that each broker, and all partition that should use the broker as leader,

Case (Leaderbroker, Topicandpartitionsforbroker)

Then find out that the current leader is not preferred, that is, the migration has occurred,
It is simple to compare directly with the leader inside the LEADERANDISR, if not equal, to indicate that there has been a migration.

Topicsnotinpreferredreplica =              topicandpartitionsforbroker.filter {                case (topicpartition , replicas) = {                  &&                  ! = leaderbroker                }              }

It will only trigger if the imbalanceratio on a broker is greater than 10% rebalance

Imbalanceratio = Totaltopicpartitionsnotledbybroker.todouble/totaltopicpartitionsforbroker

The migration process for each partition,
First the preferred broker is alive and there is no partition ongoing reassign or replica election, indicating that the process cannot be parallel, and that doing reassign is prone to conflict

//Do this check only if the broker is live and there be no partitions being reassigned currently //and preferred replica election is not in progress                  if(ControllerContext.liveBrokerIds.contains (Leaderbroker) &&controllerContext.partitionsBeingReassigned.size= = 0 &&controllerContext.partitionsUndergoingPreferredReplicaElection.size= = 0 &&!deletetopicmanager.istopicqueuedupfordeletion (topicpartition.topic) &&controllerContext.allTopics.contains (topicpartition.topic)) {onpreferredreplicaelection (Set (topicpartition),true)

Onpreferredreplicaelection

or through Partitionstatemachine, to change the state of partition.

Partitionstatemachine.handlestatechanges (partitions, onlinepartition, Preferredreplicapartitionleaderselector)

Partitionstatemachine will be analyzed separately, here only need to know, the current partition state is,onlinepartition–> onlinepartition
And a preferredreplicapartitionleaderselector, as a leaderselector strategy.

Preferredreplicapartitionleaderselector

The strategy is simply to replace leader with preferred replica

def selectleader (Topicandpartition:topicandpartition, CURRENTLEADERANDISR:LEADERANDISR): (LEADERANDISR, Seq[Int]) ={val Assignedreplicas=controllercontext.partitionreplicaassignment (topicandpartition) Val Preferredreplica= Assignedreplicas.head//take AR first replica as a preferred//Check if preferred replica is the current leaderVal Currentleader =Controllercontext.partitionleadershipinfo (topicandpartition). Leaderandisr.leaderif(Currentleader = = Preferredreplica) {//If the current leader is preferred, there's no need to do it.      Throw NewLeaderelectionnotneededexception ("Preferred replica%d is already the current leader for partition%s". Format (Preferredreplica, topicandpartition))} Else{info ("Current leader%d for partition%s was not the preferred replica." Format (Currentleader, topicandpartition) + "Trigerring preferred replica leader election")      //Check if preferred replica are not the current leader and are alive and in the ISR      if(ControllerContext.liveBrokerIds.contains (Preferredreplica) && currentLeaderAndIsr.isr.contains ( Preferredreplica)) {//determine if the current preferred replica broker is alive or not in the ISR(NewLEADERANDISR (Preferredreplica, Currentleaderandisr.leaderepoch + 1, CURRENTLEADERANDISR.ISR, Currentleaderandisr.zkversion + 1), Assignedreplicas)//Create a new LEADERANDISR}Else {        Throw NewStatechangefailedexception ("Preferred replica%d for partition". Format (Preferredreplica) + "%s are either not Ali ve or not in the ISR. Current leader and ISR: [%s] ". Format (topicandpartition, CURRENTLEADERANDISR))} }  }}

Apache Kafka Source Analysis-Autoleaderrebalanceenable