"niubi-job--a distributed task scheduling framework"----niubi-job This is even more awesome!

Niubi-job ushered in the first major optimization

　　

Niubi-job is a distributed task scheduling framework designed specifically for timed tasks, which can be used for dynamic publishing tasks and has an extremely high availability guarantee.

How many people in the middle of the night was called up to check the bug, the result is poor to the last discovery, because a certain scheduled task hangs caused a problem?

With Niubi-job, you don't have to worry about it any more!

How many people have to publish a new scheduled task, in order not to affect the online operation, only to wait until midnight to publish the application?

With Niubi-job, you can post your scheduled tasks at any time without affecting the running of your current tasks!

Isn't it exciting?

And more exciting, that is, Niubi-job released a new version of the 0.9.4.2, which is a major version of the Niubi-job history of changes, the total period of nearly one months before the completion. Although this in the middle because I changed jobs, delayed some time, but niubi-job from zero to have, the entire first version of the development took me about three weeks, and this change took one months, it can be seen how significant this change.

Next, let's see what optimizations are in this version.

　　

0.9.4.2 Maximum optimization-performance optimization

　　

Niubi-job publishes the task by uploading the jar package in the Web console, which operates the start, pause, and restart of the task on top of the Web console.

One drawback of the previous niubi-job design is that each jar package will correspond to a thread pool (the default thread pool size is 10 threads). One possible scenario is that you first released the Task-1.0.jar, which contains 1 to 101 total of 10 tasks. Later you modified the 1th task, so you upload Task-1.1.jar, this time, your task-1.0.jar corresponds to a thread pool, inside ran 9 tasks, and your Task-1.1.jar also corresponds to a thread pool, inside only ran a task.

More frightening is that you have successively put 10 tasks have been released once, until Task-1.10.jar. At this time, 10 thread pools (100 threads total) will be started, but each thread pool runs only one task (90 threads are idling), which is a great resource waste.

The first time Niubi-job was released, in order to be rolled out as soon as possible, it did not merge the thread pools of the different jar packages because there was some complexity in the class loading process.

Now that Niubi-job has formally merged the thread pool, even if you have thousands of jar packages, a single task cluster node will have only one thread pool. That is, for the above scenario, if you set the thread pool size to 20, then there will be 10 tasks running, and the other 10 threads waiting for new tasks to join. In this way, it is clear that the load capacity of the niubi-job is greatly improved.

Isn't it cool?

　　

0.9.4.2 version of the new console interface and tutorials

　　

The above is only for niubi-job performance optimization, this version also for the console interface to do some landscaping, then enjoy it together.

　　

Login with the default username admin and password 123456.

　　After logging in you can see that this is the interface of the selection mode.

　　Niubi-job support Master and slave mode, the two modes of operation interface is identical, so we take the master and slave mode to do examples, select "Master-slave" mode to enter.

　　I give each menu a note of its role, usually, our order of operation is like this.

　　1. Upload your task jar package.

　　2, click "Upload" Upload successful, you will enter the following interface.

　　3, this time, if you want to work on the task, please click on the "Schedule" button to enter the following screen.

4, usually, you only need to fill in the cron expression, select the Jar package version, click the "Execute" button, you can start a task. After you start, you can see the following interface after a few seconds of refresh. (Reminder: If your task status is always "executing", you can manually sync the task status manually by clicking the "Synchronize" button)

　　In fact, you have started a task, if you want to pause the task, or change its cron expression and restart the task, then just continue to click the "Schedule" button to do it.

　　

Some other functions of the console

　　

　　When you start the task, you go to the Operation log interface, you can see the following operation log. (Friendly reminder: This article only a start operation, the remaining two operations are my private operation produced, do not be surprised OH)

　　In addition, Niubi-job supports multi-jar versions, so when you upload only one jar, the list of all tasks is the same.

　　When you upload a 1.1 version of the jar package again, this is the list of all tasks.

The tasks in these two jar packages are the same, both Job1 and JOB2. However, in general, version 1.1 you may have made some changes to JOB1 or JOB2. This time, the All Tasks list will display all your tasks, with the unique identity of the task and the jar package name as the Federated primary key. However, in the list of actionable tasks, it will automatically follow the task's unique identity, that is, no matter how many versions of JOB1 and Job2 you upload, there is always only one Job1 and Job2 in the list of actionable tasks.

Just like this.

　　If you upload multiple versions of a jar package to a task, you can choose your jar package version when you dispatch the task. It's like the same.

　　In addition, you can see all of your cluster nodes in the cluster node list, which are the nodes that you started with the Niubi-job-cluster package to perform tasks, such as.

We started a task just now, so you will see that in the cluster node, there is a task number that has become 1, which means that it is now running a task.

　　

On the future planning of the Niubi-job

　　

Now that the Niubi-job architecture is basically stable, the next thing to do is to continue adding support for the feature, which is now added to the schedule as follows.

　　1, support the task method to pass parameters when starting a task.

2, support a single run task. (Cron expressions can be used to achieve this, but not intuitive)

3, support to start a task to specify a cluster node to run.

4, support to cluster node grouping, and can be added to a cluster node group. (The purpose of this is that if a company has two or more project groups, each project group can be assigned several cluster nodes as a group, and the task of each project group will be run on its own cluster node, without affecting the others.) Only when a cluster node of its own grouping is hung up will it temporarily borrow other grouped cluster nodes to run the task. When the nodes of this group are restored, run back to their cluster node groupings)

5, the current load balancing strategy is simply based on the number of task runs. That is, assuming there are 6 tasks and 3 nodes, each node will run 2 tasks. Later, each indicator of the task is weighted to make load balancing more intelligent. For example, there are 6 tasks, but perhaps the remaining five tasks are a few 10 seconds to run out, but there is a task to run 10 hours to run, this time only based on the number of allocation is a bit inappropriate. We will record the run time of each task and the resources it occupies to dynamically load balance.

6, operation and maintenance indicators monitoring. These include metrics for cluster nodes and tasks. For example, each cluster node CPU consumption, memory consumption and so on. For example, each task run time, whether success or not.

7, other users of the request.

　　

Conclusion

　　

Niubi-job is different from my other open source projects, this project will be continuously optimized. So if you are having trouble with a timed task, try using Niubi-job.

Finally, I hope everyone enthusiastically put forward issue and PR, my GitHub address on the left side of the blog bar. In addition, the left also has my live address, in the live, I will also answer niubi-job questions, we can also pay attention to.

Well, this is the end of the article, thank you for seeing the last.

Bow,-_-.

"niubi-job--a distributed task scheduling framework"----niubi-job This is even more awesome!