How to use Linux commands (awk, sed, bzip2, grep, wc) to make full use of multi-core CPU

Have you ever had the need to calculate a very large data (several hundred GB? Or search in it, or other operations-some operations that cannot be performed in parallel. Data experts, I am talking to you. You may have a 4-core or more multi-core CPU, but our appropriate tools, such as grep, bzip2, wc, awk, sed, etc., are all single-threaded, can you only use it? have you ever had the need to calculate a very large data (several hundred GB? Or search in it, or other operations-some operations that cannot be performed in parallel. Data experts, I am talking to you. You may have a 4-core or more multi-core CPU, but our appropriate tool, such Grep, Bzip2, Wc, Awk, SedAnd so on. they are all single-threaded and can only use one CPU core.

"How can I use these kernels "?
To enable Linux commands to use all CPU cores, we need to use the GNUParallel command to allow all of our CPU cores to perform the magic map-reduce operation on a single machine. of course, this also requires the use of rarely used ? PipesParameter (also called ? Spreadstdin). In this way, your load will be evenly distributed to each CPU, really.

BZIP2
Bzip2 is a better compression tool than gzip, but it is very slow! Don't worry about it. we have a solution to this problem.
Previous practices:
Cat bigfile. bin | bzip2 -- best> compressedfile.bz2
Now:
Cat bigfile. bin | parallel -- pipe -- recend ''-k bzip2 -- best> compressedfile.bz2
Especially for bzip2, GNU parallel is super fast on multi-core CPU. As soon as you don't care, it's done.

GREP
If you have a very large text file, you may do this before:
Grep pattern bigfile.txt
Now you can:
Cat bigfile.txt | parallel -- pipe grep 'pattern'
Or:
Cat bigfile.txt | parallel -- block 10 M -- pipe grep 'pattern'
This second usage uses ? Block10MParameter. This indicates that each kernel processes 10 million rows-you can use this parameter to adjust the number of rows of data processed by each CPU kernel.

AWK
The following is an example of using the awk command to calculate a very large data file.
General usage:
Cat rands20M.txt | awk '{s + = $1} END {print s }'
Now:
Cat rands20M.txt | parallel -- pipe awk \ '{s + =\$ 1} END {print s} \' | awk '{s + =1 1} END {print s }'
This is a bit complicated: in the parallel command ? PipeThe parameter divides the cat output into multiple blocks and assigns them to the awk call, forming many subcomputing operations. The subcomputation enters the same awk command through the second pipeline to output the final result. The first awk has three backslashes, which is required for GNUparallel to call awk.

WC
Do you want to calculate the number of lines in a file at the fastest speed?
Traditional practices:
Wc-l bigfile.txt
Now you should:
Cat bigfile.txt | parallel -- pipe wc-l | awk '{s + = $1} END {print s }'
Very clever. First, use the parallel command 'mapping' to generate a large number of wc-l calls, form a subcomputation, and finally send it to awk for summary through the pipeline.

SED
Do you want to use the sed command in a large file to perform a lot of replacement operations?
General practice:
Sed s ^ old ^ new ^ g bigfile.txt
Now you can:
Cat bigfile.txt | parallel -- pipe sed s ^ old ^ new ^ g
... Then, you can use an MPs queue to store the output to a specified file.

[Original English version: Usemultiple CPU Cores with your Linux commands]