Jstorm task scheduling Summary

Jstorm task scheduling Summary
1 Jstorm Introduction

Jstorm is an open-source distributed real-time computing system. Compared with storm, Jstorm has advantages in stability, fault tolerance, and performance. JStorm optimizes Storm performance and stability by reducing access to zookeeper, adding deserialization threads, optimizing ACK, adding monitoring content, and JAVA advantages. In short, JStorm is more powerful, more stable, and better performance than Storm. Here we mainly introduce the Scheduling Principle and Implementation of jstorm. Compared with storm, jstorm is more reasonable in Task Scheduling and more suitable for large clusters, supports advanced task scheduling functions such as custom task scheduling and dynamic resizing. Before learning about scheduling, we will briefly introduce its architecture and topology model. System Structure of jstorm.

The entire jstorm system has three different processes: nimbus, supervisor, and worker. Nimbus is mainly responsible for submitting topology and writing metadata and task scheduling to zk. supervisor starts/stops the worker process based on the scheduling results of nimbus tasks. worker is the executor of jstorm tasks, the task corresponds to a component of the topology model. Represents a topology. Spout is the message source of jstorm and is used to generate messages. bilt is the processing unit of messages, and acker is the component used to ensure that messages are not lost. The existing component has one spout, two bolts-A, two bolts-B, and one acker. These component constitute A topology where messages are transmitted through the topology tree, you can get the expected results.

Combined with the topology model and the jstorm system structure, jstorm task scheduling is to use appropriate resources in the cluster to create a reasonable number of workers on the supervisor, and allocate the component of the topology to each worker reasonably. Jstorm task scheduling has the following features:

• Ensure the average number of workers on each supervisor machine and the average number of tasks on each worker machine;
• To improve cluster stability, allocate components of the same type to different supervisor and worker as much as possible. If the components of the same type are assigned to a supervisor, if the supervisor crashes, this type of components in the topology may be lost;
• To improve performance, try to allocate the component-task with a direct message transmission relationship to the same worker. In this way, the message transmission under a worker goes through an internal channel, the transmission speed is faster;
• Supports many advanced functions. We will introduce them below;

2 Jstorm Task Scheduling

The topology is generally submitted as a parameter file. You can set the number of workers, component concurrency, and worker memory and cpu in the parameter file, as shown below. Before nimbus schedules a task, you need to determine the number of workers to be allocated.


Nimbus uses the cpu and memory information available for each sub-machine and the worker. memory. size and supervisor. slots. port. cpu. weight parameters set by the user. Based on getSupervisorPortList, the number of available ports of each supervisor and the number of available ports of the entire cluster are calculated. Therefore, the minimum value of the number of workers to be scheduled is the number of ports, topology. workers, and topology concurrency in the cluster. Determine the number of workers to be enabled for each supervisor based on the number of ports available to the supervisor, and try to ensure the average number of workers enabled on the supervisor ., Black represents the existing worker on the supervisor. Submit the topology on this cluster and five workers need to be assigned. The allocation algorithm is red. supervisor-1 first assigns worker-1, and worker assigns the order to supervisor, as shown in the label.

Determine the number of workers to be started on each supervisor, and then determine the tasks to be allocated for each worker. The principles for assigning tasks to a specific worker are as follows:

1. Assign tasks of the same component to different worker and supervisor as much as possible to ensure that the worker has the least number of tasks of the same component, and ensure that tasks of the component are connected to different supervisor or worker, this improves the stability of topology operation. The comparison function implementation is as follows: 1. traverse all workers and use the comparison function to select the appropriate worker;
   
2. Assign tasks to the supervisor as much as possible, and ensure the number of tasks on each worker is balanced. The comparison function implementation is as follows, traverse all workers and use the comparison function to select the appropriate worker;
   
3. Try to put tasks with direct message transmission on the same worker to ensure that the message is transmitted through the internal channel to ensure the transmission speed. traverse all workers and use the comparison function to select the appropriate worker;

Select the appropriate worker for the task. Filter by rule I first. If the number of workers meeting rule I is more than unique, filter by Rule ii, and the result is still not unique. Then, filter by Rule iii. In this example, we use topology_nums = 1, topology_level = 1, workers = 24, spout. parallel = 10, bolt_0.parallel = 18, acker.exe cutors = 12, and the number of clusters is 6.

• Test Objectives

① Use worker as the dimension, and try to evenly allocate worker to each supervisor;

② Ensure that tasks of the same type are evenly allocated to different worker and supervisor as much as possible. Secondly, ensure that the number of tasks of each worker and supervisor is balanced, and finally ensure that directly related tasks are put into a process;

• Test results:

The Worker is assigned to the supervisor one by one.

Tasks are allocated to work as required (there are 24 ports in total, and 12 ackers are allocated first to ensure that they can be placed in different processes and each supervisor has 2 acker; 18 bolt-tasks are assigned, the six tasks are in the same process as the acker, And the last 10 spout-tasks are allocated. They should be in the same process as the bolt-task)

3. Custom Task Scheduling

Jstorm enables custom scheduling. The ConfigExtension. setUserDefineAssignment interface is provided to you. You only need to customize the List <WorkerAssignment> data. WorkerAssignment allows the user to set the jvm parameter of the worker, the memory size of the worker, and the cpu weight. Here we use examples to illustrate how users can implement custom scheduling. The process of reading the configuration file and setting custom scheduling is shown below:

File


During task scheduling, Nimbus sets the custom scheduling priority to the highest.

4 others

• You can set the on. differ. node parameter to force tasks of the same component to be distributed to different supervisor;
• You can set the isolation. schedines. machines parameter so that a topology can be scheduled only on the specified supervisor;
• Jstorm version 9.7 finally achieves dynamic task scaling, refer to the http://gitlab.alibaba-inc.com/aloha/aloha/wikis/JStorm-Worker-Task-Scale-out-in