Preliminary discussion on batch processing framework of Spring batch

about Spring Batch

Spring batch is a lightweight batching framework, and his lightweight is relative to the cluster batch architecture of MapReduce and Spark. Although spring batch supports machine clustering, multi-threaded parallel processing is used for single machines.

We all know that MapReduce is suitable for batch processing of very large amounts of data, for a few minutes, a few 10 seconds of processing will not be placed on the MapReduce, this will only make processing time longer, because MapReduce task allocation and startup process will consume 1-2 minutes.

Spring batch is also a good choice for batches that are not particularly complex, and if a cluster like Hadoop is not available, and if a computer is multithreaded enough to handle it. (Of course, if it is enough to write a simple batch script, then the preferred batch script.) ）

However, since the official tutorial is not suitable for beginners to get started quickly, this tutorial is about two weeks to focus on the summary of the relevant content of spring batch. The order of finishing here will be as far as possible from suitable for learning and getting started. What spring batch can do.

Spring batch is used for batch processing, the earliest batch processing is used to refer to scripts, such as batch script, is to write a series of operations in a bat or sh file, a call, the script in a series of operations in order to execute each instruction.

That is, the batch is first able to contain a series of processing instructions, and can specify the order, and then the instructions are executed in the order specified. However, today is a distributed era, and a level of expansion than single-machine performance of the era, most of the excellent products are supporting parallel or cluster, spring batch in these two aspects of how to support it.

Spring batch has similar capabilities for the workflow engine, and you can configure a series of interrelated steps in the XML configuration file. You need to configure a task to define the relationship between multiple step,step and step through the next tag in the task.

At the same time, he supports multi-threaded, multi-process, multi-server processing of the same task. Next, classify the relevant data as follows: basic use

The simplest program for sping batch

In this simplest example, you will see that the step that makes up the job can be a custom tasklet, or it can be a standard itemreader, Itemprocesser, A tasklet composed of Itemwriter.

Read-write examples of different file types

Spring batch supports text files, XML files, and database reads and writes by default, providing built-in Itemreader and Itemwriter.

Configure complex business processes in conjunction with quartz or spring schedule timed triggering

quartz+spring batch tasks to trigger batches at timed intervals

Spring Schedule + Spring batch scheduled trigger- read multiple files in English

Spring Batch Multiresourceitemreader example spring batch parallel processing

SPIRNG batch parallel and extended

This article is in Chinese, written in fact, not as clear as the official website, the following is the official website address:

Scaling and Parallel processing event monitoring

Spring Batch error notification mechanism Spring Batch admin management job run

Spirng Batch Admin Tool introduction

The article is not specifically written in detail, but has given enough clues to define jobrepository as a database storage form.

Configure jobrepository-official website