Developing enterprise-level task scheduling applications based on quartz

Basic concepts and principles of quartz Scheduler

Quartz is an open-source project of opensymphony open-source organization in the task scheduling field. It is fully implemented based on Java. This project was acquired by terracotta in 2009 and is currently a project under terracotta. You can download the quartz release and source code at http://www.quartz-scheduler.org/. I used version 1.8.4 in product development. Therefore, this article is based on this version. This article not only describes how to use quartz for development, but also explains its internal implementation principles.

As an excellent open-source scheduling framework, quartz has the following features:

1. Powerful scheduling functions, such as supporting a wide variety of scheduling methods, can meet various general and special needs;
2. Flexible application methods, such as multiple combinations of tasks and scheduling, and multiple storage methods for scheduling data;
3. Distributed and cluster capabilities. After terracotta was acquired, it further improved its functionality. This article does not discuss this part for the moment.

In addition, as the default scheduling framework of spring, quartz can easily integrate with spring to implement flexible and configurable scheduling functions.

The following are some special words used in this article:

Scheduler:

Task Scheduler

Trigger:

Trigger, used to define task scheduling time rules

Job:

A task is a scheduled task.

Misfire:

Missed task scheduling that is supposed to be executed but not actually executed

Basic implementation principle of quartz Task Scheduling

Core elements

The core elements of quartz task scheduling are schedgger, trigger, and job. Trigger and job are the metadata of task scheduling, and scheduler is the Controller that actually executes the scheduling.

In quartz, trigger is an element used to define the scheduling time, that is, the time rule to execute the task. Quartz mainly provides four types of triggers: simpletrigger, crontirgger, dateintervaltrigger, and nth1_deddaytrigger. These four triggers can meet the vast majority of enterprise applications. We will further discuss the functions of four triggers in the Enterprise Application Section.

In quartz, a job is used to represent a scheduled task. There are two main types of jobs: stateless and stateful ). For the same trigger, a stateful job cannot be executed in parallel. The next trigger can be triggered only after the last trigger task is executed. A job has two main attributes: Volatility and durability. volatility indicates whether the task is persisted to the database storage, and durability indicates whether the task is retained when no trigger is associated. Both are persistent or retained when the value is true. A job can be associated with multiple triggers, but one trigger can be associated with only one job.

In quartz, schedfactory is created by the scheduler Factory: directschedulerfactory or stdschedulerfactory. The second type of factory stdschedulerfactory is widely used, because directschedulerfactory is not convenient enough to use, and many detailed manual encoding settings are required. There are three types of Scheduler: remotembeanscheduler, remotescheduler and stdscheduler. This article uses the most common stdscheduler as an example. This is also the scheduler class used by the author in the project.

Shows the relationship between the core elements of quartz:

Figure 1. Quartz core element Relationship Diagram

Thread View

In quartz, there are two types of threads: scheduler scheduling thread and task execution thread. The task execution thread usually uses a thread pool to maintain a group of threads.

Figure 2. Quartz thread View

Scheduler scheduling threads include the threads that execute regular scheduling and the threads that execute misfired trigger. The regular scheduling thread polls all the stored triggers. If a trigger needs to be triggered, that is, the next trigger time is reached, an idle thread is obtained from the task execution thread pool, executes the task associated with the trigger. The misfire thread scans all the triggers to check whether there are any misfired triggers. If yes, it will be handled separately according to the misfire policy. Describes the basic processes of these two threads:

Figure 3. Quartz scheduling thread Flowchart

We will further describe misfired trigger in the Enterprise Application Section.

Data Storage

The trigger and job in quartz must be stored for use. There are two storage methods in quartz: ramjobstore and jobstoresupport. ramjobstore stores trigger and job in the memory, while jobstoresupport stores trigger and job in the Database Based on JDBC. Ramjobstore has a very fast access speed, but because all data is lost after the system is stopped, jobstoresupport is usually used in applications.

In quartz, jobstoresupport uses a driver proxy to operate trigger and job data storage: stdjdbcdelegate. Stdjdbcdelegate implements most of the functional interfaces based on standard JDBC, but for various databases, some special processing needs to be done according to the specific implementation features, therefore, various databases need to extend stdjdbcdelegate to implement these special operations. Quartz already comes with some database extensions and can be used directly, as shown in:

Figure 4. Quartz database driver proxy

As a representative of embedded databases, Derby has become very popular recently. If you use the Derby database, you can use cloudscapedelegate in as the proxy class for trigger and job data storage.

 

Back to Top

Basic Development Process and simple examples

Build a Development Environment

Using Quartz for development is quite simple, just add the quartz-all-1.8.4.jar In the downloaded development kit to classpath. Based on my experience, for enterprise-level applications with complex task scheduling functions, it is best to import the source code of quartz into the development environment during the development stage. On the one hand, you can read the source code to understand the implementation mechanism of quartz, and on the other hand, you can extend or modify some quartz classes to implement some functions not provided by quartz.

Figure 5. Import the quartz instance project and source code

On the left is the source code imported, where org. Quartz. * is the source code of quartz. After the source code is imported, some compilation errors may occur, usually in the org. Quartz. ee. * And org.quartz.jobs. ee. * packages. The downloaded SDK has a lib directory. You can add the jar files in this directory to the compiling environment. If there is a compilation error, you can refer to the jar list on the right to search and download it online.

In the project, Com. IBM. zxn. sample. quartz is our own class package. We will use it in the following instance.

A simple instance

The quartz Development Kit has an examples directory, with 15 basic instances. We recommend that you read and practice these examples. This article only lists a small instance and introduces the basic development methods.

1. Prepare data tables for databases and quartz

   1. This article uses the IBM DB2 database: add the JDBC driver db2jcc. jar to the project;
   2. Create a quartzdb database in the database;
   3. Run/quartz-1.8.4/docs/dbtables/tables_db2_v8. SQL to create a data table. After the table is created, it is shown as follows:
   Figure 6. Quartz data table

2. Prepare the configuration file and add it to the project.

   Figure 7. instance configuration file

3. Define our own job class by implementing the job interface, as shown below:

   Figure 8. Define a task class

4. Then, the main program for task scheduling is as follows:

   In this example, we use dateintervaltrigger to schedule a task every two minutes.

   Figure 9. Main Program Implementation

5. The project structure is as follows:

   Figure 10. instance project structure

6. Run the program to view the database table and running result

   In the database, a trigger record is added to the qrtz_triggers table, as shown below:

   Figure 11. Records in the qrtz_triggers table

   A job record is added to the qrtz_job_details table as follows:

   Figure 12. Records in the qrtz_job_detailes table

   According to the running result, the task is executed every two minutes:

   Figure 13. Running result

 

Back to Top

Common applications in Enterprise Development

Some problems often occur when quartz is used for enterprise-level development. In this section, I will introduce some common problems and solutions in enterprise development based on my experience in project development.

Application 1: How to use different types of triggers

We mentioned four types of triggers in quartz: simpletrigger, crontirgger, dateintervaltrigger, and nth1_deddaytrigger.

Simpletrigger is generally used to execute a task at a specified time and how many times it is repeated. For example, it is executed once every two hours and five times it is repeated. The internal implementation mechanism of simpletrigger is to calculate the next execution time by calculating the interval, which makes it not suitable for scheduled tasks. For example, if we want to execute a task at am every day, the interval is one day if simpletrigger is used. Note that there is a problem here, that is, when there is a misfired task and the execution is resumed, the execution time is random (depending on when to execute the misfired task, such as PM of a day ). This will change the daily execution time to PM instead of the expected AM.

Crontirgger is similar to the task scheduling command crontab on Linux. It uses an expression containing seven fields to represent the time scheduling mode. For example, "0 15 10 **? * "Indicates that the task is executed at every day. Crontirgger is the most suitable option for scheduling involving weeks and months, and is even the only option in some cases. For example, "0 10 14? 3 wed indicates that the task is executed at pm every Wednesday of March. You can learn more about the meaning of each field when you use the trigger.

Dateintervaltrigger is added in Versions later than quartz 1.7. It is most suitable for scheduling tasks such as every n (1, 2, 3...) hours, every n days, and every n weeks. Although simpletrigger can implement similar tasks, dateintervaltrigger will not be affected by the misfired task we mentioned above. In addition, dateintervaltrigger will not be affected by the adjustment of DST (daylight saving time, that is, daylight saving time in China. The author once changed simpletrigger in the project to dateintervaltrigger for this reason, because if simpletrigger is used, the originally set Scheduling time will be delayed by one hour due to DST adjustment, dateintervaltrigger is not affected.

The usage of nth1_deddaytrigger is simple and clear, that is, it is used to schedule tasks on the nth day of every cycle. For example, the task is executed on the seventh day of every month.

In addition to the four triggers mentioned above, quartz also defines a calendar class (Note: org. Quartz. calendar ). This calendar is used with the trigger, but it is used to exclude tasks that are not executed. For example, a task needs to be executed on July 15, according to the trigger rules, but the calendar ar specifies that July 15 is a holiday (National Day), so the task will not be executed on this day. In general, calendar is used to exclude the scheduling of holidays, so that tasks can only be executed on workdays.

Application 2: Use statefuljob or job)

In quartz, a job is an interface that enterprise applications need to implement to define their own tasks. Basically, a task can be divided into two types: stateful and stateless. Tasks that implement the job interface are stateless by default. Quartz has another interface statefuljob. The task that implements the statefuljob interface is stateful. In the simple instance in the previous section, the samplejob we defined is a stateful task that implements the statefuljob interface. Lists the definitions of job interfaces in quartz and some built-in implementation classes:

Figure 14. Job interface definition in quartz

A stateless task is a concurrent task, that is, tasks are independent of each other and do not interfere with each other. For example, we define a trigger that is executed every two minutes, but in some cases, the next task may take three minutes to complete. In this way, when the previous task is still in the execution status, the next trigger time has arrived. For stateless tasks, they will be executed as long as the trigger time is reached, because several identical tasks can be executed concurrently. However, a stateful task cannot be executed concurrently. Only one task can be executed at a time.

In my project, some tasks need to add, delete, modify, and process data in the database. These tasks cannot be executed concurrently; otherwise, data confusion may occur. Therefore, we use the statefuljob interface. Now back to the example above, the job is executed every two minutes. If a job is executed for five minutes, what will happen to quartz? According to the trigger rule, a scheduled trigger is executed for 2nd minutes and 4th minutes respectively. However, the trigger is not triggered because the trigger is a stateful task. After the first task is completed in 5th minutes, quartz processes the 2nd and 4th minute triggers as misfired jobs. For misfired job, quartz will check how the misfire policy is set. If it is executed immediately, it will start an execution immediately. If it is waiting for the next execution, it will ignore the missed tasks, wait for the next time (6th minutes) to trigger the execution.

You can understand the differences between the two tasks and the quartz Processing Method in your project. You can select different types of tasks based on the actual situation.

Application 3: how to set the quartz thread pool and concurrent tasks

Quartz comes with the implementation of a thread pool: simplethreadpool. Class as its name. This is only a simple implementation of the thread pool and does not provide advanced features such as dynamic Spontaneous Adjustment. Quartz provides a configuration parameter: org. Quartz. threadpool. threadcount. You can set the number of threads in the thread pool during initialization, but you cannot modify it once. If the number is 10, after the number of concurrent tasks reaches 10, the triggered tasks cannot be executed and can only be executed when there are Idle threads. Therefore, some triggers may be misfire. However, it must be noted that the initial number of threads is not as large as possible. When there are too many concurrent threads, the overall performance of the system will decrease, because the system spends a lot of time on thread scheduling. Based on general experience, this value is suitable for 10-50.

Some performance-oriented thread pools are dynamically adjusted based on actual thread usage, such as the number of initial threads, the maximum number of threads, and the number of Idle threads. If you have a better thread pool in your application, you can replace simplethreadpool: org. Quartz. threadpool. Class = MyApp. greatthreadpool with the following parameters in the configuration file.

Application 4: How to Handle misfired tasks

In the quartz application, misfired job is a common situation. In general, the following causes may cause misfired job:

1) The system is restarted for some reason. Some tasks may

Misfire;

2) When a trigger is suspended (suspend) for a period of time, some tasks may be put under misfire;

3) All threads in the thread pool are occupied, causing the task to fail to be triggered, resulting in misfire;

4) when the next trigger time of a stateful task arrives, the last execution has not ended;

In order to process misfired job, quartz defines a processing policy for the trigger, mainly including the following two types:

Misfire_instruction_fire_once_now: run the command once for the misfired job;

Misfire_instruction_do_nothing: Ignore misfired job and wait for the next trigger;

We recommend that you use this configuration as a configurable option in application development so that you can dynamically configure this option for the added tirgger during use.

Application 5: how to retain ended triggers

In quartz, A tirgger is automatically deleted after the last trigger is completed. Quartz does not retain closed triggers by default, as shown in the source code of quartz below:

Figure 15. executioncomplete () source code

However, in practice, some users need to retain the previous trigger as a historical record or as a basis for creating other triggers in the future. How can I retain the ended trigger?

One solution is that the application developer maintains a data backup record and maintains a certain degree of synchronization with the records of the original quartz table. This method is cumbersome and error-prone and is not recommended.

Another method is to modify and re-compile the TRIZ trigger class to modify its default behavior. Let's take org. Quartz. simpletrigger as an example to modify if (! The code for mayfireagain () is as follows:

Figure 16. Modify the executioncomplete () source code

In addition, we need to define a new class attribute in simpletrigger: needretain, as shown below:

Figure 17. Define the new property needretain

When you define your own trigger, you can set this attribute to select whether to delete the trigger when the trigger ends. The following code is used:

Figure 18. Use the modified simpletrigger

Someone may consider implementing it by defining a new class, inheriting the org. Quartz. simpletrigger class, and overwriting the executioncomplete () method. However, this method does not work, because quartz will re-generate the simpletrigger class instance based on the trigger type during internal processing, instead of using the instance created by our own defined class. This should be a small weakness of quartz, because it blocks the ability to expand the trigger. Fortunately, quartz is open-source and can be modified as needed.

 

Back to Top

Summary

As an active open-source framework, quartz has been widely used. The powerful functions and application flexibility of quartz play a huge role in enterprise applications. This article describes how to use quartz to develop applications and discusses common problems and solutions in enterprise applications.

Original article: http://www.ibm.com/developerworks/cn/opensource/os-cn-quartz/