Kafka Consumer Partitioning Reblance algorithm

Reprint please specify original address http://www.cnblogs.com/dongxiao-yang/p/6238029.html

Recently, we need to study in detail the algorithm details of the partition calculation in the process of Kafka Reblance, searching some of the words on the internet, feeling more obscure and not easy to understand, or self-keying the source code more convenient.

The Kafka Reblance calculates part of the code as follows:

classRangeassignor ()extendsPartitionassignor with Logging {def assign (Ctx:assignmentcontext)={val valuefactory= (topic:string) = =Newmutable. Hashmap[topicandpartition, Consumerthreadid] Val partitionassignment=Newpool[string, mutable. Map[topicandpartition, Consumerthreadid]] (Some (valuefactory)) for(Topic <-ctx.myTopicThreadIds.keySet) { val curconsumers = Ctx.consumersfortopic (topic) val Curpartitions:seq[int] = ctx.partitionsfortopic (topic) Val npartspercons Umer = curpartitions.size/curconsumers.size val nconsumerswithextrapart = curpartitions.size% curconsumers.size info ("Consumer" + Ctx.consumerid + "rebalancing the following partitions:" + curpartitions + "for topic" + topic + "With consumers:" +curconsumers) for(Consumerthreadid <-curconsumers) { val myconsumerposition = Curconsumers.indexof (Consumerthreadid) assert (myconsumerposition >= 0) val Startpart = Npartspercon Sumer * myconsumerposition + myconsumerposition.min (nconsumerswithextrapart) Val nparts = Npartsperconsumer + (if ( Myconsumerposition + 1 > Nconsumerswithextrapart) 0 else 1)/*** Range-partition The sorted partitions to consumers for better locality.         * The first few consumers pick up a extra partition, if any. */        if(nparts <= 0) warn (' No broker partitions consumed by consumer thread ' + consumerthreadid + ' for topic ' +topic)Else {          For (i <-startpart until Startpart + nparts) {val partition = curpartitions (i) info (consumerthreadid+ "Attempting to claim partition" +partition)//record the partition ownership decisionVal Assignmentforconsumer =partitionassignment.getandmaybeput (consumerthreadid.consumer) Assignmentforconsumer+ = (topicandpartition (topic, partition)Consumerthreadid)} }      }    }

  def getpartitionsfortopics (topics:seq[string]): mutable. Map[string, seq[int]] = {          = = =      topicandpartitionmap._1=  Topicandpartitionmap._2      Debug ("Partition assignment of/brokers/topics/%s is%s". Format (topic, Partitionmap  )    partitionMap.keys.toSeq.sortWith ((s,t) and S <  t)    }  }}

def getconsumerspertopic (group:string, Excludeinternaltopics:boolean): mutable. Map[string, List[consumerthreadid]] ={val dirs=NewZkgroupdirs (Group) Val Consumers=getchildrenparentmaynotexist (Dirs.consumerregistrydir) Val Consumerspertopicmap=Newmutable. Hashmap[string, List[consumerthreadid]] for(Consumer <-consumers) {val TopicCount= Topiccount.constructtopiccount (Group, Consumer, This, Excludeinternaltopics) for(topic, consumerthreadidset) <-topiccount.getconsumerthreadidspertopic) { for(Consumerthreadid <-consumerthreadidset) consumerspertopicmap.get (topic) match { CaseSome (curconsumers) =consumerspertopicmap.put (Topic, Consumerthreadid:: curconsumers) Case_ =consumerspertopicmap.put (topic, List (Consumerthreadid))}} }     for(topic, consumerlist) <-Consumerspertopicmap) consumerspertopicmap.put (topic, Consumerlist.sortwith (s,t) + S < t)) Consumerspertopicmap}

The calculation process is mainly implemented by the above highlighted code section, for example, a 10-partition topic, the same group has three Consumerid for AAA,CCC,BBB consumers

1 by the latter two pieces of code, get Consumerid list and partition partition list are already sorted, so

Curconsumers= (AAA,BBB,CCC)

Curpartitions= (0,1,2,3,4,5,6,7,8,9)

2

NPARTSPERCONSUMER=10/3 =3

nconsumerswithextrapart=10%3 =1

3 Assuming the current client ID is AAA

myconsumerposition= Curconsumers.indexof (AAA) =0

4 Calculating the partition range

startpart= 3*0+0.min (1) = 0

Nparts = (if (0 + 1 > 1) 0 else 1) =3+1=4

So the AAA corresponding partition number is [0,4], that is, the 0,1,2,3 front four partitions

Similarly, the BBB corresponds to the myconsumerposition=1, corresponding to the partition 4,5,6 the middle of the three partitions

CCC corresponds to myconsumerposition=2, corresponding to the last three partitions of 7,8,9.

Kafka Consumer Partitioning Reblance algorithm