Development of enterprise-level task scheduling application based on Quartz

Original address: https://www.ibm.com/developerworks/cn/opensource/os-cn-quartz/

Quartz is an open source project for Opensymphony Open source organization in the field of task scheduling, based entirely on Java implementations. As an excellent open source scheduling framework, Quartz has the features of powerful, flexible application and easy integration. This paper analyzes the basic realization principle of Quartz framework, describes the basic method of application Quartz development application through some concrete examples, and discusses the common problems and solutions in enterprise application.

Quartz Basic concepts and principles Quartz Scheduler Open Source Framework

Quartz is an open source project for Opensymphony Open source organization in the field of task scheduling, based entirely on Java implementations. The project was acquired by Terracotta in 2009 and is currently a project of Terracotta. Readers can download Quartz's release version and its source code to the http://www.quartz-scheduler.org/site. The author uses version 1.8.4 in product development, so the content of this article is based on this version. This article not only introduces how to use Quartz for development, but also explains its internal realization principle.

As an excellent open-source scheduling framework, Quartz has the following features:

1. Powerful scheduling functions, such as supporting a wide variety of scheduling methods, can meet various conventional and special needs;
2. Flexible application methods, such as support tasks and scheduling of a variety of combinations, to support the scheduling of data storage methods;
3. Distributed and cluster capabilities, Terracotta acquisition after the original function based on the further improvement. This article does not discuss this part of the content

In addition, as the spring default scheduling framework, Quartz is easy to integrate with spring to enable flexible and configurable scheduling capabilities.

Here are some of the special terms used in this article, which declares:

Scheduler

Task Scheduler

Trigger

triggers, which define task scheduling time rules

Job

Tasks, which are scheduled tasks

Misfire:

a task scheduled to be executed but not actually executed.

Basic implementation principle of Quartz task scheduling

Core elements

The core element of Quartz task scheduling is scheduler, trigger and job, where trigger and job are the metadata of task scheduling, scheduler is the controller that actually executes the dispatch.

In Quartz, trigger is the element used to define the scheduling time, which is the time rule to perform the task. There are four types of Trigger:simpletrigger,crontirgger,dateintervaltrigger, and Nthincludeddaytrigger, which are mainly available in Quartz. These four types of trigger can meet most of the needs of enterprise applications. We will further discuss the functions of the four trigger in the Enterprise Application section.

In Quartz, the job is used to represent the task being dispatched. There are two main types of job: stateless (stateless) and stateful (stateful). For the same trigger, a stateful job cannot be executed in parallel, and the next execution can be triggered only after the last triggered task has been executed. There are two main properties of the job: volatility and durability, where volatility indicates whether the task is persisted to the database store, and durability indicates whether the task is persisted without a trigger association. Both are persisted or persisted when the value is true. A job can be associated with multiple trigger, but a trigger can only associate one job.

In Quartz, Scheduler is created by the Scheduler Factory: Directschedulerfactory or Stdschedulerfactory. The second factory stdschedulerfactory used more, because the directschedulerfactory is not easy to use, need to make a lot of detailed manual coding settings. There are three main types of Scheduler: Remotembeanscheduler, Remotescheduler and Stdscheduler. This article takes the most commonly used stdscheduler as an example to explain. This is also the scheduler class used by the author in the project.

The relationships between Quartz core elements are as follows:

Figure 1. Quartz core element diagram

Threads View

In Quartz, there are two types of threads, Scheduler dispatch threads and task execution threads, where task execution threads typically maintain a set of threads using a thread pool.

Figure 2. Quartz Threads View

Scheduler has two main scheduling threads: The thread that executes the regular schedule, and the thread that executes the misfired trigger. The regular dispatch thread polls all trigger stored, and if there is a trigger that needs to be triggered, that is, when the next trigger is reached, the task execution thread pool Gets an idle thread to perform the task associated with that trigger. Misfire threads are scanned for all trigger to see if there are misfired trigger, and if any, are handled separately according to the misfire policy. Describes the basic flow of these two threads:

Figure 3. Quartz Scheduling thread Flowchart

For misfired Trigger, we will describe further in the Enterprise Application section.

Data storage

The trigger and jobs in Quartz need to be stored to be used. There are two ways to store Quartz: Ramjobstore, Jobstoresupport, where Ramjobstore is to store trigger and jobs in memory, and Jobstoresupport is based on JDBC Trigger and The job is stored in the database. Ramjobstore access is very fast, but because all of the data is lost after the system is stopped, the jobstoresupport is used in the usual application.

In Quartz, Jobstoresupport uses a driver agent to manipulate trigger and job data storage: stdjdbcdelegate. Stdjdbcdelegate implements most of the standard JDBC-based functional interfaces, but for a variety of databases it is necessary to do some special processing according to their specific implementation, so the various databases need to expand stdjdbcdelegate to achieve these special processing. Quartz has brought in some of the database extension implementations that can be used directly, as shown in:

Figure 4. Quartz Database Driver Agent

As a representative of the embedded database, Derby has been very popular recently. If you use the Derby database, you can use Cloudscapedelegate in as the proxy class for the trigger and job data stores.

Back to top of page

Basic development process and simple example

Build the development environment

Developing with Quartz is fairly straightforward, just add the Quartz-all-1.8.4.jar from the download development package to the CLASSPATH. According to the author's experience, it is best to import the Quartz source code into the development environment in the development stage for enterprise application with complex task scheduling function. On the one hand, you can read the source code to understand the implementation mechanism of Quartz, on the other hand, you can extend or modify some of the Quartz classes to achieve some Quartz not yet provide functionality.

Figure 5. Quartz example Project and source code import

The left is the source code after the import, wherein org.quartz.* is the source of quartz. There may be some compilation errors after importing the source code, usually in the org.quartz.ee.* and org.quartz.jobs.ee.* packages. There is a Lib directory in the download development package that allows readers to add jar files from the directory to the compilation environment. If there is a compilation error, readers can refer to the list of jars on the right, and go to the web to search for downloads.

Com.ibm.zxn.sample.quartz in the project is our own class package, which we'll use in the example below.

A simple instance

There is a examples directory in the Quartz development package, which has 15 basic instances. Readers are advised to read and practice these examples. This article only lists a small example of the basic development method.

1. Prepare data tables for databases and Quartz

   1. This article uses the IBM DB2 database: Add the JDBC driver Db2jcc.jar to the project;
   2. Create a new library in the database quartzdb;
   3. Execute/quartz-1.8.4/docs/dbtables/tables_db2_v8.sql, create a data table, and the table is built as follows:
   Figure 6. Quartz Data Sheet

2. Prepare the configuration file and add it to the project

   Figure 7. Instance configuration file

3. Define our own task class by implementing the job interface, as follows:

   Figure 8. Defining Task Classes

4. Then, implement the Task Scheduler for the main program, as follows:

   In this example, we use Dateintervaltrigger to implement a task schedule that executes every two minutes.

   Figure 9. Implementing the main program

5. After completion, the project structure is as follows:

   Figure 10. Instance Project structure diagram

6. Run programs to view database tables and run results

   Database, a trigger record is added to the Qrtz_triggers table as follows:

   Figure 11. Records in the Qrtz_triggers table

   A JOB record is added to the Qrtz_job_details table as follows:

   Figure 12. Records in the Qrtz_job_detailes table

   From the running results, the task is executed every two minutes:

   Figure 13. Run results

Back to top of page

Common applications in enterprise-level development

There are a number of issues that are frequently encountered when applying Quartz for enterprise-level development. In this section, based on my own experience in project development, the author introduces some common problems in enterprise development and the usual solutions.

Application one: How to use different types of Trigger

Earlier we mentioned four types of Trigger:simpletrigger,crontirgger,dateintervaltrigger in Quartz, and Nthincludeddaytrigger.

Simpletrigger is generally used to implement tasks at regular intervals, and how many times, such as executing every 2 hours and repeating 5 times. The Simpletrigger internal implementation mechanism calculates the next execution time by calculating the interval time, which makes it unsuitable for scheduling scheduled tasks. For example, we want to 1:00am every day, and if we use Simpletrigger, the time interval is one day. Note that there is a problem here: when there is a misfired task and resumes execution, the execution time is random (depending on when the misfired task is performed, such as a day's 3:00pm). This results in a daily execution time that becomes 3:00pm, not the 1:00am we originally expected.

Crontirgger is similar to the Task Scheduler command crontab on LINUX, which uses an expression of 7 fields to represent the time-scheduling method. For example, "0 15 10 * *?" * "indicates the daily 10:15am to perform the task. For schedules that involve weeks and months, Crontirgger is the most suitable, and even in some cases, the only option. For example, "0 10 14?" 3 WED "represents March of every Wednesday afternoon 14:10pm perform the task. Readers can learn more about the meaning of each field when they use the trigger specifically.

Dateintervaltrigger is a version of Quartz 1.7, which is best suited for scheduling tasks like every n (1, 2, 3 ...) hours, every n days, and every n weeks. Although Simpletrigger can achieve similar tasks, Dateintervaltrigger will not be affected by the misfired tasks we have mentioned above. In addition, Dateintervaltrigger is not affected by DST (daylight saving time, China daylight saving time) adjustment. For this reason, the author has changed the Simpletrigger of the project to Dateintervaltrigger, because if Simpletrigger is used, the scheduled time will be one hours earlier or delayed due to DST adjustment, and Dateintervaltrigger is not affected by this.

The use of Nthincludeddaytrigger is simple and clear, that is, for the day ordinal of every other cycle, for example, the 3rd day of each month to perform the specified task.

In addition to the 4 types of Trigger,quartz mentioned above, a Calendar class (note, Org.quartz.Calendar) is also defined. This Calendar is used with Trigger, but their role is reversed, and it is used to exclude tasks from being executed. For example, according to the rules of Trigger, you need to perform a task on October 1, but the Calendar specifies that October 1 is a holiday (National day), so the task will not be executed on this day. In general, the Calendar is used to schedule tasks that exclude holidays, so that tasks are performed only on weekdays.

Application Two: Use stateful (statefuljob) or stateless tasks (job)

In Quartz, job is an interface, and enterprise applications need to implement this interface to define their own tasks. Basically, the task is divided into stateful and stateless two kinds. The task that implements the job interface defaults to a stateless state. There is another interface statefuljob in the Quartz. The task of implementing the Statefuljob interface is stateful, and in a simple instance of the previous section, the samplejob we define is the stateful task that implements the Statefuljob interface. Lists the definition of the Job interface in Quartz and some of its own implementation classes:

Figure 14. Job interface definition in Quartz

Stateless tasks generally refer to tasks that can be concurrent, that is, the tasks are independent and do not interfere with each other. For example, we define a trigger that executes every 2 minutes, but in some cases a task may take 3 minutes to complete, so that the next trigger time is reached when the previous task is still in the execution state. For stateless tasks, as long as the trigger time is up, it is executed because several identical tasks can be executed concurrently. However, for stateful tasks, it is not possible to execute concurrently, only one task can be executed at a time.

In the author's project, some tasks require the processing of data in the database to be increased and censored. These tasks cannot be executed concurrently, or they can cause data confusion. So we use the Statefuljob interface. Now back to the example above, the task executes every 2 minutes, and if a task executes for 5 minutes to complete, what will Quartz do with it? According to the rules of Trigger, the 2nd and 4th minutes will have a predetermined trigger execution, but because there is a stateful task, it will not actually be triggered. At the 5th minute when the first task is completed, Quartz will handle the two triggers for the 2nd and 4 minutes as the misfired job. For misfired Job,quartz will see how its misfire policy is set, if it is executed immediately, it will start once execution, if it is waiting for the next execution, will ignore the missed task, and wait for the next time (that is, 6 minutes) to trigger execution.

Readers can experience the difference between the two tasks in their own project and the Quartz, and choose different types of tasks depending on the situation.

Application three: How to set Quartz thread pool and concurrency task

The implementation of a thread pool has been brought into the Quartz: Simplethreadpool. Class as its name, this is just a simple implementation of the thread pool and does not provide advanced features such as dynamic self-tuning. Quartz provides a configuration parameter: Org.quartz.threadPool.threadCount, which can set the number of threads in the thread pool at initialization time, but cannot be modified once set. Assuming that this number is 10, after the concurrent task reaches 10, then the task that has triggered will not be executed, only wait for the idle thread to be executed. Therefore, some trigger may be misfire. It must be noted, however, that the number of initial threads is not as large as possible. When there are too many concurrent threads, the overall performance of the system degrades, because the system spends a lot of time on thread scheduling. According to general experience, this value is more appropriate in 10-50.

For some performance-focused thread pools, dynamic adjustments are made based on actual thread usage, such as the number of initial threads, the maximum number of threads, the number of idle threads, and so on. Readers in the app, if there is a better thread pool, you can replace SimpleThreadPool:org.quartz.threadPool.class = MyApp with the following parameter in the configuration file. Greatthreadpool.

Application four: How to handle misfired tasks

In Quartz applications, the misfired job is often encountered. In general, the following reasons may cause misfired job:

1) The system is restarted for some reason. During the period between system shutdown and restart, some tasks may

Be misfire;

2) Trigger is suspended (suspend) for a period of time, some tasks may be misfire;

3) All threads in the thread pool are occupied, causing the task to not be triggered to execute, resulting in misfire;

4) The status of the task at the next trigger time arrives, the last execution has not ended;

In order to deal with misfired Job,quartz the processing strategy is defined for trigger, there are two main types:

Misfire_instruction_fire_once_now: Execute once for misfired job;

Misfire_instruction_do_nothing: Ignore misfired job, wait for next trigger;

It is recommended that the reader use this setting as a configurable option in app development, allowing users to dynamically configure this option for Tirgger that have been added during usage.

Application five: How to keep the closed Trigger

In Quartz, a tirgger is automatically deleted after the last trigger is completed. Quartz does not retain the closed trigger by default, as shown in the following Quartz source code:

Figure 15. Executioncomplete () Source code

However, in real-world applications, some users need to retain their previous trigger as a historical record, or as a basis for creating additional trigger later. How to keep the end of the trigger?

One approach is for the application developer to maintain a copy of the data backup record and to maintain a certain synchronization with the records of the Quartz original table. This method is actually more cumbersome to operate, and error-prone, not recommended to use.

Another option is to modify the default behavior of Quartz by modifying and recompiling the trigger class. We use Org.quartz.SimpleTrigger as an example to modify the code in the If (!mayfireagain ()) section of the above code as follows:

Figure 16. Modify Executioncomplete () source code

In addition, we need to define a new class attribute in Simpletrigger: Needretain, as follows:

Figure 17. Defining a new property Needretain

When you define your own trigger, you can choose whether to delete trigger at the end of the trigger when you set this property. As shown in the following code:

Figure 18. Using the modified Simpletrigger

One might consider implementing by defining a new class and then inheriting the Org.quartz.SimpleTrigger class and overriding the Executioncomplete () method. However, this approach does not work because Quartz internally regenerates instances of the Simpletrigger class based on the type of trigger, rather than instances created with our own defined classes. This should be a small shortcoming of Quartz because it has blocked the ability to expand trigger. Fortunately Quartz is open source, we can make changes as needed.

Back to top of page

Summary

As an open-source framework of current vitality, Quartz has been widely used. Quartz's powerful features and application flexibility have played a huge role in enterprise applications. This article describes how to use Quartz to develop applications, and discusses common problems and solutions in enterprise applications.

Reference Learning

• Visit the official website of Quartz.
• Stay tuned for DeveloperWorks technical activities and webcasts.
• Access the DeveloperWorks Open source zone for rich how-to information, tools and project updates, and the most popular articles and tutorials to help you develop with open source technology and use them with IBM products.

Discuss

• Joining the DeveloperWorks Chinese community, the DeveloperWorks community is a professional social networking community for global IT professionals who can provide community functions such as blogs, bookmarks, wikis, groups, contacts, sharing, and collaboration.
• Join the IBM software download and technology Exchange Group to participate in online communication.

Development of enterprise-level task scheduling application based on Quartz