Java Distributed task Scheduling xxl-job

Little Chenbo main 2017-11-30 21:11:56

preface to the Broken language

In the stand-alone application period, the task scheduling is generally based on spring schedule and integrated quartz to achieve, when the system developed into distributed services, the application of multiple instances, the task will appear many times the problem, many times our task does not need to run many times. There are many solutions, the simplest and most brutal of which is the application switch can be set. The second is centralized word task management scheduling. Of course, quartz also has cluster mode, but it is not intuitive to control based on API. The following introduces a centralized distributed task scheduling framework, which can easily solve the problem of distributed task scheduling.

One, Xxl-job introduction

Xxl-job is a lightweight distributed task scheduling framework, its core design goal is to develop quickly, learn simple, lightweight, easy to expand. Now open the source code and access to a number of companies online product line, open box is used. Features are as follows

1, simple: Support through the Web page to face the task of crud operations, simple operation, one minute to start;

2, Dynamic: Support the dynamic modification of task status, Suspend/restore tasks, and terminate the operation of the task, immediate effect;

3, the dispatch center Ha (center type): The dispatch uses the center type design, "The Dispatch center" is based on the cluster quartz realization, may guarantee the dispatch center ha;

4, Actuator ha (distributed): Task distributed execution, task "actuator" to support the cluster deployment, to ensure that the task execution ha;

5, Task failover: the implementation of cluster deployment, the Task routing policy select "Failover" when the scheduling failure will be smooth switch actuator to failover;

6, Consistency: "Dispatch center" through DB lock to ensure the consistency of the cluster distributed scheduling, a task scheduling will trigger only one execution;

7, custom task parameters: Support Online configuration scheduling task to participate in the immediate effect;

8, scheduling thread pool: Scheduling system multithreading trigger scheduling operation to ensure that the scheduling of accurate implementation, not blocked;

9, Elastic expansion shrink Capacity: Once the new actuator machine on-line or offline, the next schedule will be reassigned tasks;

10, mail alarm: Mission failure to support the mail alarm, support the configuration of multiple mailing address mass alarm mail;

11, State monitoring: Support real-time monitoring task progress;

12, rolling execution log: Support to view the scheduling results online, and support the rolling way to view the complete execution log of the actuator output in real time;

13, GLUE: Provide the web IDE, support the online development task logic code, dynamic Publishing, real-time compilation effective, omitting the deployment of the online process. Supports 30 versions of historical version backtracking.

14, Data encryption: Dispatching center and the communication between the actuator data encryption, improve the security of scheduling information;

15, task dependencies: Support the configuration subtask dependencies, when the parent task is completed and successful execution will trigger the execution of subtasks once, multiple subtasks separated by commas;

16, push the MAVEN Central warehouse: will be pushed to the latest stable version of the MAVEN central warehouse, user-friendly access and use;

17, the task registration: The executor will automatically register the task periodically, the dispatch center will automatically find the registered task and trigger the execution. At the same time, also support the manual entry actuator address;

18, routing Strategy: Actuator Cluster Deployment provides a wealth of routing strategies, including: The first, the last, polling, random, consistent hash, the most infrequently used, the last time the longest unused, failover, busy transfer, etc.

19, the operation of the report: support real-time viewing of operational data, such as the number of tasks, scheduling times, the number of actuators, as well as scheduling reports, such as scheduling date distribution map, scheduling successful distribution map, etc.;

20, Script task: Support in glue mode to develop and run script tasks, including Shell, Python and other types of scripts;

21, congestion processing strategy: Scheduling too dense actuator too late to deal with the processing strategy, the strategy includes: Single machine serial (default), discard subsequent scheduling, coverage before scheduling;

22, failure processing strategy, scheduling failure of the processing policy, policies include: Failure alarm (default), failure retry;

23, the fragmented broadcast task: When the Actuator cluster deployment, Task routing strategy select "Fragmented broadcast" case, a task scheduling will broadcast trigger cluster all actuators to perform a task, according to the piecewise parameters of the development of fragmentation tasks;

24, Dynamic fragmentation: Fragmentation broadcast task to the dimensions of the implementation of fragmentation, support dynamic expansion of the actuator cluster to dynamically increase the number of fragmented, collaborative business processing, in the large volume of data business operations can significantly enhance the task processing capacity and speed.

25, Event Trigger: In addition to "cron Mode" and "task-dependent" trigger task execution, support event-based trigger task. The dispatch Center provides API services that trigger a single execution of tasks, which can be triggered flexibly according to business events.

second, Quick start

The project address is: Https://github.com/xuxueli/xxl-job

2.1 Initializing the Dispatch database

Please download the project source code and extract, get "Dispatch database initialization SQL Script" and execute it, normally should generate 16 tables.

The scheduled database initialization SQL script location is:

/xxl-job/doc/db/tables_xxl_job.sql

The dispatch center supports the cluster deployment, in which the nodes must connect the same MySQL instance;

If MySQL is the master from, the dispatch center cluster node must be forced to take the main library;

2.2 compiling source code

Extract the source code, according to the MAVEN format will be imported into the IDE, using MAVEN to compile, the source structure is as follows:

Xxl-job-admin: Dispatch Center

Xxl-job-core: Public Reliance

Xxl-job-executor: Executor Sample sample (select the appropriate version executor to use directly, or refer to it and change the existing project to the executor)

: Xxl-job-executor-sample-spring:spring version, through the spring container management actuator, more general, recommend this way;

: Xxl-job-executor-sample-springboot:springboot version, through the Springboot management actuator;

: Xxl-job-executor-sample-jfinal:jfinal version, through the jfinal management actuator;

2.3 Configuration Deployment "dispatch Center"

Dispatch Center project: xxl-job-admin

Role: The unified management task scheduling platform scheduling task, responsible for triggering the scheduling execution, and provide task management platform.

Step One: Dispatch center configuration:

Dispatch Center configuration file Address:

/xxl-job/xxl-job-admin/src/main/resources/xxl-job-admin.properties

Dispatch Center Configuration Content Description:

### Dispatch center JDBC Link: Link address please keep the same address as the dispatch database created in section 2.1

Xxl.job.db.driverclass=com.mysql.jdbc.driver

Xxl.job.db.url=jdbc:mysql://localhost:3306/xxl-job?useunicode=true&characterencoding=utf-8

Xxl.job.db.user=root

Xxl.job.db.password=root_pwd

### Alarm Mailbox

Xxl.job.mail.host=smtp.163.com

Xxl.job.mail.port=25

Xxl.job.mail.username=ovono802302@163.com

Xxl.job.mail.password=asdfzxcv

Xxl.job.mail.sendfrom=ovono802302@163.com

xxl.job.mail.sendnick= "Task scheduling platform Xxl-job"

### login Account

Xxl.job.login.username=admin

xxl.job.login.password=123456

### Dispatch Center Communication token, non-null enabled

xxl.job.accesstoken=

Step Two: Deploy the project:

If this configuration is done correctly, the project can be compiled into a war package and deployed to Tomcat. Dispatch Center Access Address: http://localhost:8080/xxl-job-admin (the address executor will be used, as a callback address), after the login run interface as shown in the following figure

The dispatch center project has been successfully deployed to this point.

Step Three: dispatch center cluster (optional):

The dispatch center supports cluster deployment and improves the availability of scheduling system.

The only requirement for cluster deployment is to ensure that each cluster node configuration (DB and login account, etc.) is aligned. The dispatch center distinguishes different clusters through DB configuration.

The dispatch center can assign a domain name to the cluster by Nginx load balancing at the time of cluster deployment. The domain name can be used for access on one hand, and can be used to configure executor callback addresses on the other hand.

2.4 Configuring the Deploy Executor project

Executor Project: xxl-job-executor-sample-spring (such as a new executor project, refer to the configuration steps for the sample executor project;)

Role: Be responsible for receiving the dispatch and execution of the dispatch center, deploying the executor directly, or integrating the executor into an existing business project.

Step One: Maven dependencies

Confirm that the "Xxl-job-core" maven dependency is introduced in the Pom file;

Step Two: Actuator configuration

Actuator configuration, configuration file address:

/xxl-job/xxl-job-executor-samples/xxl-job-executor-sample-spring/src/main/resources/ Xxl-job-executor.properties

Actuator configuration, description of configuration content:

### xxl-job Admin Address List: Dispatch center deployments and addresses: If the dispatch center cluster deployment exists multiple addresses, it is separated by commas. The executor will use this address for "Executor heartbeat Registration" and "task result callback."

Xxl.job.admin.addresses=http://127.0.0.1:8080/xxl-job-admin

### xxl-job Executor Address: Executor "AppName" and addressing information configuration: AppName Actuator heartbeat Enrollment Group by; address information is used for "dispatch center requests and triggers tasks" and "Executor registration." Actuator default port is 9999, actuator IP default is NULL to automatically obtain IP, multiple network card can manually set the specified IP, manually set IP will bind host. When deploying multiple actuators on a stand-alone machine, be careful to configure different actuator ports;

Xxl.job.executor.appname=xxl-job-executor-sample

xxl.job.executor.ip=

xxl.job.executor.port=9999

### xxl-job Log path: The disk location where the executor runs the log file store, which requires read and write access to the path

xxl.job.executor.logpath=/data/applogs/xxl-job/jobhandler/

### Xxl-job, access token: Executor communication token, non-null enabled

xxl.job.accesstoken=

Step Three: Actuator Component Configuration

Actuator component, configuration file address:

/xxl-job/xxl-job-executor-samples/xxl-job-executor-sample-spring/src/main/resources/ Applicationcontext-xxl-job.xml

Actuator component, description of configuration content:

<!--configuration 01, jobhandler Scan Path: Jobhandler--> in automatic scanning container

<context:component-scan base-package= "Com.xxl.job.executor.service.jobhandler"/>

<!--configuration 02, Actuator Excutor Configuration: Actuator Core Configuration-->

<bean id= "Xxljobexecutor" class= "Com.xxl.job.core.executor.XxlJobExecutor" "init-method=" "Start" destroy-method= " Destroy ">

<!--actuator ip[selected], automatically get--> for null

<property name= "IP" value= "${xxl.job.executor.ip}"/>

<!--actuator port number [must]-->

<property name= "Port" value= "${xxl.job.executor.port}"/>

<!--actuator appname[Optional], NULL to turn off autoenrollment-->

<property name= "AppName" value= "${xxl.job.executor.appname}"/>

<!--Actuator Registration center address [optional], blank to turn off autoenrollment-->

<property name= "adminaddresses" value= "${xxl.job.admin.addresses}"/>

<!--actuator log path [Required]-->

<property name= "LogPath" value= "${xxl.job.executor.logpath}"/>

<!--access token, non-empty to match checksum [optional]-->

<property name= "Accesstoken" value= "${xxl.job.accesstoken}"/>

</bean>

Step Four: Deploy the Executor project:

If the above configuration has been done correctly, the executor project can be compiled and deployed, and the system provides three sample samples of the executor, one of which can be deployed in the following manner.

Xxl-job-executor-sample-spring: Project compilation is packaged into a war package and deployed to Tomcat.

Xxl-job-executor-sample-springboot: Project compiled into a springboot type of executable jar package, command to start;

The executor project has been deployed to the end.

Step five: Actuator cluster (optional):

The actuator supports cluster deployment, improves the availability of scheduling systems, and improves task processing capabilities.

The only requirement for cluster deployment is to ensure that the configuration item "xxl.job.admin.addresses/Dispatch center Address" for each executor in the cluster is consistent, and that the executor performs automatic registration of the actuator according to the configuration.

2.5 Develop the first task "Hello World"

This example takes the task of creating a new glue mode (Java) run mode as an example. For more detailed configuration of tasks, see "chapter III: Task Details." ("Glue mode (Java)" Execution code is hosted to the dispatch Center for online maintenance, more simple and lightweight than "bean mode task" needs to be deployed on the Executor project development

Prerequisites: Verify that the dispatch center and executor projects have been successfully deployed and started;

Step One: Create a new task:

To login to the dispatch center, click the "New Task" button shown in the image below to create a new example task. Then, refer to the task parameter configuration in the screenshot below and click Save.

Step Two: "Glue mode (Java)" Task development:

Please click on the "GLUE" button on the right of the task to enter the "GLUE Editor Development Interface", see figure below. The task of the Glue mode (Java) run mode has initialized the sample task code by default, that is, print Hello world. (The task of the Glue Mode (Java) run mode is actually a section of Java class code that inherits from Ijobhandler, runs in the executor project, and uses the @resource/@Autowire to inject other services in the executor)

step three: Trigger execution:

Click the "Execute" button on the right side of the task to manually trigger a task execution (usually by configuring a cron expression to start a task dispatch).

Step four: View the log:

Please click on the "Log" button on the right side of the task to view the task log from the task log interface. In the task log interface, you can view historical scheduling records for the task, as well as task scheduling information, execution parameters, and execution information for each schedule. Running tasks click on the "Execute Log" button on the right to go to the log console to view the live execution log.