The task scheduling algorithm of Storm source reading notes

Overview of 3 Scheduler

• EventScheduler: Distribute the available resources in the system evenly to the topology that need the resources, in fact, it is not absolutely uniform, the follow-up will elaborate
• DefaultScheduler: Similar to Evenetscheduler, but will be the other topology unnecessary resources collected again, and then Eventscheduler
• IsolationScheduler: The user can define this topology machine resource, and when Storm allocates it, these topology are assigned preferentially to ensure that the machine assigned to the topology is only for this topology service

Defaultscheduler

• Call the cluster needsSchedualerTopologies method to get the task assignment topologies
• Begin processing each topology separately
  • Call the cluster getAvailableSlots method to get the resources available to the current cluster, <node,port> return it as a collection, assign a value toavailable-slots
  • Obtain the current topology executor information and convert it into a <start-t ask-id,end-task-id> set deposit all-executors , according to topology calculation executors information, using the compute-executors algorithm, will explain later
  • Then call the Eventscheduler get-alive-assigned-node+port->executors method to get the topology already obtained resources, return the <node+port,executor> form of the collection to deposit alive-assigned , why to calculate the current topology the allocated resources situation instead of compute all the allocated resources in the cluster? , guess it might be useful when you're rebalance a task.
  • It then calls the slot-can-reassign slots information in the alive-assigned to determine which slots can be redistributed into the variablecan-reassigned
  • The resources that are available are made up of available-slots can-reassigned two parts
  • Next, calculate the total number of slots that the current topology can use total-slots--to-use : Min (numworker number of topology, available-slots + can-reassigned )
  • If total-slots--to-use > is currently assigned the number of slots, the method is called to bad-slots calculate the slot that can be freed
  • Call the cluster freeSlots method to release the computed Bad-slot
  • The last call to Eventscheduler is schedule-topologies-evenly allocated
  • Continue to the next topology

主要流程梳理: Calculates the executor information for the current topology (which will be used on allocation), and computes the assignable and deallocated resource, which is the current cluster idle resource.

Eventscheduler

The Eventscheduler scheduling algorithm is less than the default one that computes the reallocation of resources, and distributes them directly using the idle slots in the supervisor, no longer in detail.

Eventscheduler and Defaultscheduler scheduling examples:

These two scheduling mechanisms in general the scheduling results are basically consistent, so together to see:

Cluster initial state

Next we submit 3 topology

Topology	Worker number	Executer number	Task number
T-1	3	8	16
T-2	5	10	10
T-3	3	5	10

1. Submit T-1

• The sort-slots algorithm handles the available slots with the result {[S1 6700] [S2 6700] [S3 6700] [S4 6700] [S1 6701] [S2 6701] [S3 6701] [S4 6701] [s1 6702] [s2 6702] [S3 6702] [S4 6702] [S1 6703] [S2 6703] [S3 6703] [S4 6703]}
• The executor list obtained by the compute-executors algorithm is: {[1 2] [3 4] [5 6] [7 8] [9 10] [11 12] [13 14] [15 16]};注：格式为[start-task-id end-task-id]，共8个worker，第一个包含2个task，start-task-id为1，end-task-id为2，所以记为[1 2]，后面依次类推...compute-executors算法会在下一篇博客中详解
• The distribution status of 8 executor on 3 workers is [3,3,2]
• The result of the assignment is:
  • {[1 2] [3 4] [5 6]}-[S1 6700]
  • {[7 8] [9] [s2 6700]}
  • {[[] [] [S3 6700]

After allocation, the cluster status is:

2. Submit T-2

• Available slots after sort-slots: {[S1 6701] [S2 6701] [S3 6701] [S4 6700] [S1 6702] [S2 6702] [S3 6702] [S4 6701] [S1 6703] [S2 6703] [S3 6703] [S4 6702] [S4 6703]}
• Comput-executors Calculated Executor list: {[1 1] [2 2] [3 3] [4 4] [5 5] [6 6] [7 7] [8 8] [9 9] [10 10]}
• The distribution of 10 executor on 5 workers is [2,2,2,2,2]
• The result of the assignment is:
  • {[1 1] [2 2]}, [S1 6701]
  • {[3 3] [4 4]}, [S2 6701]
  • {[5 5] [6 6]}, [S3 6701]
  • {[7 7] [8 8]}, [S4 6700]
  • {[9 9] [ten]}--[S1 6702]

After allocation, the cluster status is:

3. Submit T-3

• Sort-slots after slot list: {[S1 6703] [S2 6702] [S3 6702] [S4 6701] [S2 6703] [S3 6703] [S4 6702] [S2 6704] [S3 6704] [S4 6703] [s 4 6704]}
• The list of executor received after Compute-executors is: {[1 2] [3 4] [5 6] [7 8] [9 10]}
• The distribution of 5 executor on 3 workers is: [2,2,1]
• The result of the assignment is:
  • {[1 2] [3 4]}, [S1 6703]
  • {[5 6] [7 8]}, [S2 6702]
  • [9], [S3 6702]

After allocation, the cluster status is:

, this task scheduling mode is not absolutely uniform, S1 is already full load operation, and S4 just use a slots.

The algorithm used in this article, such as comput-executors、sort-slots、slots-can-reassign、bad-slots、sort-slots等will be devoted to the next blog post on

The task scheduling algorithm of Storm source reading notes