A preliminary understanding of Springbatch

Springbatch is a batch-processing framework that, as a spring component, provides the conditions for processing batches by using spring's dependency injection.

What is batch processing?

In modern enterprise applications, in the face of complex business and massive data, in addition to through a large number of human-computer interaction interface for various processing, there is a kind of work, do not need manual intervention, only need to read the bulk data regularly, and then complete the corresponding business processing and archiving. This type of work is "batching". For example, the recent financial projects in the Matchmaking function, do not need manual intervention, the borrower and the lender automatically matching matchmaking, complete the borrowing function.

As can be seen from the above description, batch processing applications have the following characteristics:

• • The amount of data is large, less million, more than billions of orders of magnitude.
  • No manual intervention is required and is automatically completed by the system according to the configuration.
  • Time-related, such as executing once per day or once a month.

At the same time, batch processing applications are clearly divided into three parts:

• • Read data, data may come from files, databases, or message queues
  • Data processing, such as billing for telecom support systems
  • Write data, write output to file, database or message queue, etc.

Therefore, from the system architecture, you should focus on the transaction granularity of batch applications, log monitoring, execution, resource management (especially in the case of concurrency). From the system design, we should focus on the decoupling of data read-write and business processing, improve reusability and testability.

Second, Springbatch's business scenarios are:

• Periodic commit batch Processing
• To process a task in parallel
• Message-driven Application grading processing
• Massively parallel batch Processing
• Restart after manual or scheduled use task fails
• Sequential execution with dependent steps (using workflow-driven extensions)
• Skip partial records while processing
• Batch transactions: For small batches or for scenarios where stored procedures/scripts are used

III. structure diagram of Springbatch

The Springbatch uses a three-tier architecture, with three tiers for applications, cores, and basic services.

　　

Refinement of the architecture

　　

　　

Application layer: Contains all batch jobs, managing programmer-defined code through the Spring framework.

Core layer: Contains the core classes required for batch launch and control, such as Joblauncher, job, and step.

The basic service layer: The application layer and the core layer are built on the basic service layer that is the infrastructure layer IQ, the infrastructure layer provides common read (Itemreader), write (Itemwriter) and services (such as Retrytemplate: Retry module, can be used by the application layer and the core layer).

Also, we need to know that Spring batch is a batch application framework, not a scheduling framework. It only focuses on issues related to batch tasks, such as transactions, concurrency, monitoring, execution, etc., and does not provide the corresponding scheduling function. Therefore, if we want the batch processing task to be performed periodically, it can be implemented with a mature scheduling framework such as Quartz.

A preliminary understanding of Springbatch