Jstorm and Storm Source Analysis (iii)--scheduler, scheduler

Scheduler, as a Storm scheduler, is responsible for allocating available resources to the topology.
Storm provides a IScheduler interface that allows users to customize the scheduler by implementing the interface.
It is defined as follows:

Public interface IScheduler {    //receives the storm configuration of the current Nimbus as a parameter, doing some initialization work    void Prepare (Map conf);    /**     * The method to actually perform the task assignment, which is called when the task assignment is Nimbus.     * Parameters are topologies, cluster: The former contains static information of all topology in the current cluster,     * Cluster contains the topology of the running state information, such as the user to customize the scheduling logic of all the resources required,     * Supervisor information, all slots currently available     *, as well as allocation of objects, etc., according to topologies and cluster information, can be scheduled to assign tasks *    /void Schedule (topologies topologies, Cluster Cluster);}

The way to really choose which scheduler to assign to topology is mk-assignments.
The Mk-assignments method is defined and interpreted as follows:

As you can see from the code above, if you call the Getforcedscheduler method in Inimbus and return the non-null IScheduler, the IScheduler instance is returned, and if the user implements the custom IScheduler, and is configured in Storm.yaml, the user-defined IScheduler is returned, and if neither is implemented, the assignment of the task is performed with the default scheduler Defaultscheduler. Now we only care about Defaultscheduler.
The definition and interpretation of Defaultscheduler are as follows:

;;D Efaultscheduler is the default scheduler for storm, if the user does not specify their own implementation of the scheduler;; Storm will use the Scheduler for Topology assignment.;;D The Efaultscheduler implements the IScheduler interface (NS Backtype.storm.scheduler.DefaultScheduler  (: Use [backtype.storm util Config] )  (: Require [Backtype.storm.scheduler.EvenScheduler:as Evenscheduler])  (: Import [Backtype.storm.scheduler IScheduler topologies            Cluster topologydetails workerslot schedulerassignment            Evenscheduler executordetails] )  (: Gen-class    : Implements [Backtype.storm.scheduler.IScheduler]))

;; The Default-schedule method mainly calculates all the available slot resources in the current cluster; and determine whether the slot resource currently assigned to the topology needs to be reassigned;; Use this information to allocate resources for the newly submitted topology (Defn default-schedule [^topologies topologies ^cluster Cluster];; Call cluster's Needsschedulingtopologies method to get the topology collection required for task scheduling;;  The Needsschedulingtopologies method definition is as shown in FN1. (Let [Needs-scheduling-topologies (. needsschedulingtopologies cluster topologies)];; The purpose of this block of code is to process each topology that requires task scheduling (Doseq [^topologydetails topology needs-scheduling-topologies;; Get topology-id:let by calling GetId [Topology-id (. getId topology);; Call cluster's Getavailableslots method to get all the available slot resources in the current cluster;;            and convert it to the <node,port> set assigned to the Available-slots variable. ;;                               The Getavailableslots method defines the Available-slots (->> (. Getavailableslots cluster) as shown in fn2 below. (Map # (vector (. Getnodeid%)           (. Getport%)))) ;; Call Getexecutors to get all executor information for topology;;    and convert it to <start-task-id,end-task-id> collection all-executors (->> topology                         . Getexecutors (Map # (vector (. Getstarttask%)                             (. Getendtask%))) set);; Call the Get-alive-assigned-node+port->executors method of Evenscheduler;; Calculates the task information that is currently assigned to the current topology, and saves the <[node,port],executors> information to the alive-assigned variable alive-assigned (evenscheduler/get-ali          Ve-assigned-node+port->executors cluster topology-id);; Alive-executors (->> alive-assigned vals (apply concat) set);; Call the Slots-can-reassign method to judge the alive-assigned slot information;;           Select which slot sets can be reassigned and save to Can-reassign-slots. ;; The Slots-can-reassign method definition is as shown in Fn3: Can-reassign-slots (slots-can-reassign cluster (keys alive-assigned));;                                  Calculates the total number of slots that the current topology can use, taking the lesser of the following two values as Total-slots-to-use total-slots-to-use (min (. Getnumworkers topology)          (+ (Count can-reassign-slots) (Count Available-slots))) ;; Used to determine if the number of total-slots-to-use is greater than the current assignedNumber of slots,;; or the number of executors that are running is not equal to all executors;;          The Bad-slots method is called to calculate all the slots that can be freed. ;;          The specific definition of the Bad-slots method is as shown in Fn4. Bad-slots (if (or (> Total-slots-to-use (Count alive-assigned)) (not= alive-executors all-e                        xecutors)) (Bad-slots alive-assigned (Count all-executors) total-slots-to-use) [])]]      ;; Call cluster's Freeslots method to release the previously computed Bad-slots (. freeslots cluster bad-slots);; Call Evenscheduler's Schedule-topologies-evenly method to distribute the resources in the system evenly to topology (Evenscheduler/schedule-topologies-evenly ( Topologies. {Topology-id topology}) cluster))))

FN1:

/** * Gets all the topology that need to be dispatched and returns the */public list<topologydetails> Needsschedulingtopologies as a collection (topologies topologies) {    list<topologydetails> ret = new arraylist<topologydetails> ();    For (Topologydetails topology:topologies.getTopologies ()) {        if (needsscheduling (topology)) {            Ret.add ( topology);        }    }    return ret;}

FN2:

Gets all available slot resources based on the supervisor information and encapsulates them in Workerslot, returning the public list<workerslot> Getavailableslots as a collection ( Supervisordetails supervisor) {    set<integer> ports = this.getavailableports (supervisor);    list<workerslot> slots = new arraylist<workerslot> (Ports.size ());    for (Integer port:ports) {        slots.add (new Workerslot (Supervisor.getid (), port));    }    return slots;}

FN3:

;; This method will filter the incoming slots resources and select the slots which can still be used to make up the new collection;; Filtering method: First determine whether the slot node information exists in the blacklist of the cluster; If not, continue to determine if the slot port information is in the list of all available ports for the node corresponding to the supervisor;; If it is, it indicates that the slot can continue to be used (defn slots-can-reassign [^cluster Cluster Slots]  (->> slots      (the filter        (FN [[node Port]] (          if-not (. isblacklisted cluster node) (            If-let [Supervisor (. Getsupervisorbyid cluster node)]              (. Contains (. getallports supervisor) (int port)))))))              

FN4:

;; This method is used to calculate what is no longer needed for a topology already allocated resource;; Existing-slots: Allocated resources (assigned to topology), it is a <[node,port],executors> collection;; All executor of num-executors:topology (including allocated and unassigned);; Num-workers:topology total number of slots available for use (defn-bad-slots [existing-slots num-executors num-workers];; Determine if the num-workers is 0. If it is, it means that there is currently no slot available for the topology, and an empty collection is returned (if (= 0 num-workers) ' ();; Defines the distribution collection and the Keepers collection, distribution collection is generated by calling the Integer-divided method;;    What is actually done is to distribute the num-executors evenly into the num-workers. ;;       Keepers set to an empty collection (let [Distribution (Atom (integer-divided num-executors num-workers)) Keepers (Atom {})] ;; For each entry in the Existing=slots, calculate the executor-count of its object;; The Executor-count is then used as the key to get the value from the distribution collection that was previously computed. If the obtained value is greater than 0;; It means that at least one worker has a executor-count executor allocation, and that the allocation information is maintained and not updated.;      At this point, the <[node,port],executors> information is put into keepers, and the corresponding value of the Executor-count in the distribution is reduced by one. (Doseq [[Node+port executor-list] Existing-slots:let [Executor-count (Count Executor-list)] "When (POS? (Get @distribUtion executor-count 0) (swap! Keepers Assoc node+port executor-list) (swap! distribution update-in [ex Ecutor-count]));; Removes the allocations that are recorded in the keepers that need to continue to be maintained from the existing-slots. If the slot information exists after removal,;      Indicates that these slots can be freed, converted to a collection of Workerslot objects, and returned.                   (->> @keepers keys (apply Dissoc existing-slots) keys (Map (FN [[node port]] (Workerslot node port)))))

Note: Learn Li Ming and other teachers storm source analysis and Chen Min-min and other teachers storm technical insider with big data practice notes collation.
Please pay attention to the following QR code for technical exchanges:

Jstorm and Storm Source Analysis (iii)--scheduler, scheduler