Storm Series (vii) scheduler-Scheduler [Defaultscheduler]

Storm default Task Scheduler. The implementation is as follows:

1	(defn –prepare [ this conf])
2	(defn   –schedule   [ this   ^ topologies   topologies   ^ cluster   cluster]
3	(default-schedule topologies cluster))

Default-schedule

Method Prototypes:

1	(defn   Default - schedule   [ ^ topologies   Topologies   ^ cluster   cluster])

Method Description:

1. Call the Needsschedulingtopology method of the cluster object to get the topology collection that needs to be scheduled for the task.
2. Call cluster's Getavailableslots method to get the slot resources available for the current cluster (supervisor ports that are not in use in the cluster) and convert to <node,port> collection (available-slots).
3. Converts the Executordetails collection in topology to the <start-task-id,end-task-id> collection into the all-executors.
4. Call the Get-alive-assigned-node+port->executors method to get the current topology already allocated resource condition, return <node+port,executors> collection ( alive-assigned).
5. Call the Slots-can-ressign method to determine the slot information for the alive-assigned, and select the slot set where it can be reassigned and save it to the can-ressign-slots variable.
6. Calculates the number of all slots that the current topology can use, topology sets the number of workers and the current number of available-slots plus the minimum value of the Can-ressign-slots data ( Total-slots-to-use).
7. Determines whether the number of total-slots-to-use is greater than the number of slots currently allocated (alive-assigned), or, if greater than, calls the Bad-slots method to calculate all slots that may be freed.
8. Call cluster's Freeslots method to release the previously computed bad-slots.
9. Call Eventscheduler's Schedule-topologies-evenly method to distribute the topology in the system evenly.

Scheduling Flowchart:

slots-can-reassign

Functionality: Filters out resources that can continue to be used from resources that are already assigned to the current topology.

Function Prototypes:

1	(Defn Slots-can- reassign [ ^ Cluster Cluster slots])

Method Description:

1. The parameter slots is a collection of allocated slots resources,<node,port>.
2. Filtering the incoming slots, filtering method: Choose to determine whether the slots node information exists in the blacklist of the cluster, if not, continue to determine whether the slot port is in the corresponding supervisor node of all the available ports list, If you are in the slots, you can continue to use it.

Bad-slots

Function: Calculates which of the resources that a topology has allocated is no longer needed.

Function Prototypes:

1	(defn- bad -slots [Existing- Slots num- executors num-workers])

Parameter description:

Existing-slots: A collection of resources,<[node,port],executors> that have been assigned to topology.

All the executor of the num-executors:topology (both allocated and unassigned).

Num-workers:topology the total number of slots that can be used.

Method Description:

1. Determines whether the num-workers is 0, if it represents a slots that is currently not available for the topology and returns an empty collection.
2. Define the distribution, keepers collection, call the Integer-divided method to distribute the num-executors evenly to num-worksers in the distribution collection, the result set format < Executor-count,worker-count>, Executor-count represents the number of executor that a single worker is assigned, and Worker-count indicates how many such workers The Keepers collection defaults to empty.
3. The executor-count is calculated for each item in the Existing-slots, and then according to the distribution collection, and the Executor-count is the key to get the value, if the obtained value is greater than 0, meaning that there is such an allocation, then the <[ The node,port],executors> information is placed in the keepers, and the corresponding value of the Executor-count in the distribution is reduced by one.
4. Removing the required continuation of the keepers record from the existing-slots, and if the slot information exists after removal, the slots can be freed and converted to a collection of Workerslot objects to return.

Storm Series (vii) scheduler-Scheduler [Defaultscheduler]