Step into the enterprise-level batch processing framework--springbatch

Springbatch is a lightweight, fully spring-oriented batch processing framework that can be applied to a large number of data processing systems at the enterprise level. Spring batch can provide a large number of repeatable data processing functions, including logging/tracking, transaction management, job processing statistics restart, skip, and resource management and other important functions. It enables business people to focus on the development of their core business, and takes repetitive, time-consuming work to the system for automated processing. such as data pouring, exporting, copying data and so on. This article introduces how Springbatch works with a small example of a simple file copy. First look at the relevant core code and configuration:

<?xml version= "1.0" encoding= "UTF-8"? ><beans xmlns= "Http://www.springframework.org/schema/beans" xmlns: Batch= "Http://www.springframework.org/schema/batch" xmlns:xsi= "Http://www.w3.org/2001/XMLSchema-instance" xmlns: tx= "Http://www.springframework.org/schema/tx" xmlns:aop= "Http://www.springframework.org/schema/aop" xmlns: context= "Http://www.springframework.org/schema/context" xsi:schemalocation= "http://www.springframework.org/ Schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd http://www.springframework.org/ Schema/tx http://www.springframework.org/schema/tx/spring-tx-3.0.xsd HTTP://WWW.SPRINGFRAMEWORK.ORG/SCHEMA/AOP Http://www.springframework.org/schema/aop/spring-aop-3.0.xsd Http://www.springframework.org/schema/context http ://www.springframework.org/schema/context/spring-context-2.5.xsdhttp://www.springframework.org/schema/batch Http://www.springframework.org/schema/batch/spring-batch.xsd "default-autowire=" byname "><bean id=" Joblauncher "class=" ORg.springframework.batch.core.launch.support.SimpleJobLauncher "><property name=" jobrepository "ref=" Jobrepository "/></bean><bean id=" jobrepository "class=" Org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean "></bean><bean id=" TransactionManager "class=" Org.springframework.batch.support.transaction.ResourcelessTransactionManager "/> <batch:job id= "Iconvjob" ><batch:step id= "Iconvstep" ><batch:tasklet transaction-manager= " TransactionManager "><batch:chunk reader=" Iconvitemreader "writer=" Iconvitemwriter "processor=" Iconvitemprocessor "commit-interval=" 1 "/></batch:tasklet></batch:step></batch:job>< Context:property-placeholder location= "classpath:files.properties"/><bean id= "Iconvitemreader" class= " Com.inetpsa.batch.iconv.IconvItemReader "><property name=" input "value=" File:${input.file} "/>< Property Name= "CharSet" value= "UTF-8"/></bean><bean id= "IconvitemwriteR "class=" Com.inetpsa.batch.iconv.IconvItemWriter "><property name=" Output "value=" File:${output.file} "/> <property name= "CharSet" value= "UTF-8"/></bean><bean id= "Iconvitemprocessor" class= " Com.inetpsa.batch.iconv.IconvItemProcessor "/></beans>

Springbatch's core configuration is reflected in this configuration file, each batch will contain one (or more jobs), each job defines the steps we want to complete the job, that is, the job of step, The completion of each step requires the support of the corresponding persistence mechanism, and jobrepository is the identity of the provider of the persistence mechanism. Reflected in the configuration file, the role of Joblauncher is to open a job by an external controller call, When a job is opened, it enters the substantive step execution phase, and each step is executed by Itemreader first reading the data and then returning it to itemprocessor for processing and then returning it to itemwriter for output. We can show each step to define Itemreader,itermprocessor and itemwriter. As defined in the configuration file, we have developed three classes for each of the corresponding base classes to implement custom read, process, and output. Here's a look at the specific implementation.

public class Shellitemreader implements itemreader<inputstream> {private String input;private inputstream item = Nu Ll;public InputStream Read () throws Exception, Unexpectedinputexception, ParseException, nontransientresourceexception {if (This.item = = null) {This.item = This.input = = null? System.in:new URL (this.input). OpenStream (); return this.item;} return null;} public void SetInput (String input) {this.input = input;}}

The Shellitemreader class reads the files that are configured in the configuration file, and then returns the input stream to shelliterprocessor for processing:

public class Shellitemprocessor implements Itemprocessor<inputstream, inputstream> {private list<string> Command;public InputStream Process (InputStream item) throws Exception {final Processbuilder PB = new Processbuilder (this. command);p B.redirecterrorstream (TRUE); final process process = Pb.start (); Ioutils.copy (item, Process.getoutputstream ()); ioutils.closequietly (item); ioutils.closequietly (Process.getoutputstream ()); return Process.getinputstream ();} public void SetCommand (list<string> command) {this.command = command;}}

shellitemprocessor processes the input stream and returns it to the Shellitemwriter for output:

public class Shellitemwriter implements itemwriter<inputstream> {private String output;public void Setoutput ( String output) {this.output = output;} public void write (list<? extends inputstream> items) throws Exception {outputstream os = system.out;if (this.output ! = null) {final URL url = new URL (this.output), if (Url.getprotocol (). Equals ("file")) {OS = new FileOutputStream (url.getpat h ());} else {OS = Url.openconnection (). Getoutputstream ();}} For (final InputStream is:items) {ioutils.copy (was, OS); ioutils.closequietly (is);} ioutils.closequietly (OS);}}

In the case of database or file read data, Shellitemreader's read () operation reads one record at a time and then hands it to shellitemprocessor. When the number of records processed reaches the configured Commit-interval value, the processed data is handed to Shellitermwriter for a single data. We also configured the transaction for each step in order to roll back when the program went wrong. These are the simple processes that springbatch executes.

Step into the enterprise-level batch processing framework--springbatch