Linux parallel rsync copy n multiple files

Take a look at the comparison chart, the image on the left is a copy of the common Rsync directory, the right figure is the use of the parallel tool and launched multiple rsync

centos6.5 Installation Parallel

 #!/bin/bash# Install parallel on CentOS  6  .# assumes is root. Prefix  w /sudo  if   not.cd /etc/yum . repos.d/< Span style= "color: #000000;" >#  wget  http:// download.opensuse.org/repositories/home:tange/centos_centos-5/home:tange.repo  wget  http:// download.opensuse.org/repositories/home:/tange/centos_centos-6/home:tange.repo  yum  install  parallel 

Start rsync copy file [copy from/data/to/hadoop/]

[Ten~]$DF-hfilesystem Size used Avail use%mounted on/dev/mapper/volgroup-lv_root 50G6.1G 41G -% /tmpfs 32G 12K 32G1%/dev/SHM/DEV/SDA1 485M 39M 421M9% /Boot/dev/mapper/volgroup-lv_home5.5G 140M5.1G3% /Home/dev/mapper/data-apps 12T5+216.7T -% /Data/dev/mapper/bigdata-hadoop 12T3.5T8.3T -%/hadoop

If you use common rsync, plus a lot of small files, one night only passed more than 1T, using the parallel tool 1 hours probably passed 500G

CD/datafind'mkdir-p/hadoop/{//}; rsync-a {}/hadoop/{}'

Linux parallel rsync copy n multiple files