[Use the linux concurrent command parallel to speed up text file processing]-it turned out to be a pitfall.

1, see an article above oschina how to use multi-core CPU to accelerate your Linux Command & mdash; awk, sed, bzip2, grep, wc and other http: wwwoschinanetnews45477use-cpu-cores-linux-commands systems use Redhat5parall 1. see an article on oschina.

How to use multi-core CPU to accelerate your Linux commands-awk, sed, bzip2, grep, wc, etc.

Http://www.oschina.net/news/45477/use-cpu-cores-linux-commands

The system uses Redhat 5.

Official website of parallel:

Http://www.gnu.org/software/parallel/

Use the source of opensuse for installation.

Http://software.opensuse.org/download.html? Project = home % 3 Atange & package = parallel

For CentOS CentOS-6, run the following command as root user root:

cd /etc/yum.repos.d/wget http://download.opensuse.org/repositories/home:tange/CentOS_CentOS-6/home:tange.repoyum install parallel

For CentOS CentOS-5, run the following command as root user root:

cd /etc/yum.repos.d/wget http://download.opensuse.org/repositories/home:tange/CentOS_CentOS-5/home:tange.repoyum install parallel

The wc-l command showed that the computing time was not increased when the number of CPUs was 4, but it was slow.

Calculates the number of lines in 1 GB of files. it takes 10 s to use wc-l and parallel.

The relationship between MB files is also 10 times.

Summary:

There is still a difference between theory and practice. There is no advantage in testing 4 cup machines.

It takes more than 10 times to slow down. It seems that hadoop is more reliable to process large files.