GNU Parallel: Parallel execution of Linux commands

A very interesting question to see on stack overflow today is:

You need a script to process a text file, but this text file can be very large, because of the problem of processing the script, you want to be able to process only a few lines of the text file, like Xargs, and can not use split to split the file, and does not allow the generation of temporary files.

If temporary files are allowed, then it is not difficult to use bash script, which can be implemented using the GNU Parallel tool if temporary files cannot be used. The method is implemented as follows:

Cat Giantfile.txt | Parallel-j 8--pipe-l 50000 Import_script

-j 8: Represents the number of concurrent jobs and does not want parallel execution to be set to 1. You can also run a job for each CPU core by default, without the-J option.

--pipe: Reads a piece of data from stdin and assigns each piece of data to each jobs

-l N: reads up to n rows of data at a time

Here's a parallel tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

This is a translation version: http://my.oschina.net/enyo/blog/271612

Here are some examples of parallel used with other Linux commands: http://www.vaikan.com/use-multiple-cpu-cores-with-your-linux-commands/

GNU Parallel: Parallel execution of Linux commands