Implementation of unified job scheduling monitoring and control in distributed Architecture (I.)

The job trigger mechanism based on quartz can solve the time dependence. But what we often encounter is the dependency between the job, for example, job A is successful to execute job B. At the same time we expect to have a log record of the execution of the job, if the job execution fails, can notify us in the way of alarm (mail/SMS), further or have an interface to configure the job, query job performance. Especially now many systems are divided into many modules, especially the application of micro-service architecture, if multiple modules have a separate job trigger mechanism, is no doubt a waste, also is not conducive to unified management, so the birth of the proposed need to have a unified job scheduling monitoring management module.

The scheme is based on quartz to do time triggering. A JMS is used to communicate between the Management module (job-admin) and the managed module. There are multiple benefits based on JMS: 1,job-admin and monitored modules are decoupled, job-admin are not directly interacting with the monitored modules, and are independent of each other. 2, the reliability of information transmission, if the job-admin send information, but the monitored module has been shut down or down, the message will remain in the queue, will not be lost, wait until the monitored module back to normal, then continue to process the information. 3, asynchronous communication, the issue of information does not depend on the execution of the job. Job-admin sends a message, returns immediately and does not wait for the message to be processed. 4, the same monitored module can be distributed deployment, each deployment of each module to monitor the same queue, ensuring that the job implementation of the uniqueness and reliability.

In addition to having a job SDK or job-client, the SDK needs to be integrated into modules that need to be managed, which will not affect the original logic and minimize intrusion. For each integrated module to have a separate JMS queue, such as a module, we define QUEUE_A, Job-admin send the trigger message to the QUEUE_A, the SDK in module A listens to the queue and gets the information from it. At the same time also define queue_callback, all monitored module public this queue, the monitored module will send three kinds of information to the queue, respectively, is the job has begun, job has been completed, job failure. Job-admin Monitor Queue_callback, will receive the information in storage or alarm notification.

Job-admin work: Our common jobs are divided into days and hours per minute, and job dependencies we only focus on the dependencies between jobs that are executed by day. Each job should have a run time threshold, or alert if the threshold is not successful. If job B relies on job A, job a feedback exception, or timeout does not have feedback, then job B does not execute, and can then be manually triggered to perform job B after fixing the problem.

Job-admin to have three tables to store, job_tpl,job_instance,job_relation

Job_instance record Job Instance execution

Id,jobname,status,createtime,starttime,finishedtime,failedtime,errormessage (varchar1000)

JOB_TPL Record Job definition

Name,create_time,group,queue,status,cron_expression,desc,type (daily, non-daily type), Time_string,max_run_time

Job_relation Record Job dependencies

Id,child_name,parent_name

Each job_instance must have its own State created,started,finished,failed,none,expired

The Job-client SDK package is based on spring AOP and annotation. The job method to be executed by each monitored module is annotation, and when the monitored module starts, it initializes all the Annotaiton methods. When the SDK receives a JMS message, it finds the corresponding job method, executes the method through reflection, and job-admin the execution through the callback JMS notification.
JMS format
Call JobName, Jobid

Call Back Jobname,jobid,status,errormessage

Concrete realization, let's.