Storm-Kafka [interface implementation] 4-1: zkcoordinator: ZK Coordinator

Background: You need to have a basic understanding of ZK and Kafka.

Topic of this chapter: detailed process of zkcoordinator

Package COM. mixbox. storm. kafka; import Org. slf4j. logger; import Org. slf4j. loggerfactory; import COM. mixbox. storm. kafka. trident. globalpartitioninformation; import Java. util. *; import static COM. mixbox. storm. kafka. kafkautils. taskid;/***** zkcoordinator Coordinator ** @ author Yin Shuai */public class zkcoordinator implements partitioncoordinator {public static final logger log = loggerfactory. getlogger (zkcoordinator. class); spoutconfig _ spoutconfig; int _ taskindex; int _ totaltasks; string _ topologyinstanceid; // each partition corresponds to a partition manager Map <partition, partitionmanager> _ managers = new hashmap (); // The cached listlist <partitionmanager> _ cachedlist; // The Last refresh time long _ lastrefreshtime = NULL; // refresh frequency: millisecond int _ refreshfreqms; // dynamic partition connection dynamicpartitionconnections _ connections; // dynamic brokersreaderdynamicbrokersreader _ reader; zkstate _ state; Map _ stormconf; /***** @ Param connections * dynamic partition connection * @ Param stormconf * storm configuration file * @ Param spoutconfig * storm sput configuration file * @ Param state * For zkstate connect * @ Param taskindex * task * @ Param totaltasks * Total task * @ Param topologyinstanceid * instance id of the topology */Public zkcoordinator (dynamicpartitionconnections connections, map stormconf, spoutconfig, zkstate state, int taskindex, int totaltasks, string tasks) {This (connections, stormconf, spoutconfig, state, taskindex, totaltasks, triggers, buildreader (stormconf, spoutconfig);} public zkcoordinator (dynamicpartitionconnections connections, map stormconf, spoutconfig, zkstate state, int taskindex, int totaltasks, string handler, dynamicbrokersreader reader) {_ spoutconfig = spoutconfig; _ connections = connections; _ taskindex = taskindex; _ totaltasks = totaltasks; _ topologyinstanceid = response; _ stormconf = stormconf; _ state = State; zkhosts brokerconf = (zkhosts) spoutconfig. hosts; _ refreshfreqms = brokerconf. refreshfreqsecs * 1000; _ reader = reader;}/*** @ Param stormconf * @ Param spoutconfig * @ return */Private Static dynamicbrokersreader buildreader (MAP stormconf, spoutconfig) {zkhosts hosts = (zkhosts) spoutconfig. hosts; return New dynamicbrokersreader (stormconf, hosts. brokerzkstr, hosts. brokerzkpath, spoutconfig. topic) ;}@ overridepublic list <partitionmanager> getmymanagedpartitions () {If (_ lastrefreshtime = NULL | (system. currenttimemillis ()-_ lastrefreshtime)> _ refreshfreqms) {refresh (); _ lastrefreshtime = system. currenttimemillis ();} return _ cachedlist;}/*** simple refresh behavior **/void refresh () {try {log.info (taskid (_ taskindex, _ totaltasks) + "refreshing Partition Manager connections"); // obtain all the partition information globalpartitioninformation brokerinfo = _ reader. getbrokerinfo (); // obtain the list of all partitions of the task <partition> mine = kafkautils. calculatepartitionsfortask (brokerinfo, _ totaltasks, _ taskindex); // obtain the partition set <partition> curr = _ managers of the current task. keyset (); // construct a set <partition> newpartitions = new hashset <partition> (mine); // remove all the newpartitions in the new partition. removeall (curr); // set <partition> deletedpartitions = new hashset <partition> (curr); // deletedpartitions. removeall (mine); log.info (taskid (_ taskindex, _ totaltasks) + "deleted partition managers:" + deletedpartitions. tostring (); For (partition ID: deletedpartitions) {partitionmanager man = _ managers. remove (ID); man. close ();} log.info (taskid (_ taskindex, _ totaltasks) + "new partition managers:" + newpartitions. tostring (); For (partition ID: newpartitions) {partitionmanager man = new partitionmanager (_ connections, _ topologyinstanceid, _ state, _ stormconf, _ spoutconfig, ID); _ managers. put (ID, man) ;}} catch (exception e) {Throw new runtimeexception (E);} _ cachedlist = new arraylist <partitionmanager> (_ managers. values (); log.info (taskid (_ taskindex, _ totaltasks) + "finished refreshing");} @ overridepublic partitionmanager getmanager (partition) {return _ managers. get (partition );}}

1: zkcoordinator implements the partitioncoordinator interface.

Package COM. mixbox. storm. kafka; import Java. util. list; /*** @ author Yin Shuai */public interface partitioncoordinator {/*** get list of my managed partitions {partitionmanager} * @ return */list <partitionmanager> getmymanagedpartitions (); /*** @ Param go to getmanager * @ return */partitionmanager getmanager (partition) based on the specified partition );}

The first method obtains all the partitionmanagers.

The second method obtains a partition manager based on a specific partition.

For partitionmanager, refer to another blog in this space:

Storm-Kafka [interface implementation] 4-2: partitionmanager: Partition Manager