Bash script for Batch Job parallelization

When running a job in Linux, you often encounter the following situations: a large number of jobs need to be run, and the time required to complete each job is not very long. if we run these jobs in serial mode, it may take a long time. If we run these jobs in parallel mode, the running time can be greatly reduced. furthermore, most of the current computers are in multi-core architecture. to make full use of their computing capabilities, parallel computing is required. summarize the information on the Internet. With the Bash script, you can use the following methods to parallel Batch jobs:

When running a job in Linux, you often encounter the following situations: a large number of jobs need to be run, and the time required to complete each job is not very long. if we run these jobs in serial mode, it may take a long time. If we run these jobs in parallel mode, the running time can be greatly reduced. furthermore, most of the current computers are in multi-core architecture. to make full use of their computing capabilities, parallel computing is required. to sum up the information seen on the Internet, we can use the following methods to achieve Batch Job parallelization using the Bash script. note: The process and thread are not distinguished, and parallel and concurrent tasks are not distinguished.

1. Use the GNU paralle Program

Parallel is a program specially used by GNU for parallelization. It is suitable for Simple Batch Job parallelization. you do not need to write scripts when using parallel. Simply add parallel to the original command. therefore, if you can use paralle to parallelize your job, use it first. for more information about paralle, see its official documentation.

2. The simplest method of Parallelism: & + wait

Using Bash's background running & wait functions, you can achieve the simplest Batch Job parallelization.

The following code takes about 10 seconds for serial execution: