Method and efficiency test of efficient synchronization of data--side packing edge compression edge transmission Edge decompression 20150105

Sometimes in the backup or synchronization with a large number of files (such as a few GB or dozens of GB of database directory, log directory), the direct SCP takes longer, although the first compression and re-transmission can be decompressed, the amount of data transmitted is really reduced, but compression and decompression will also take a lot of time , the overall effect is not satisfactory, yesterday night whim, due to previous experience in streaming video-on-demand projects, if you can look like HD video only need to download the video file metadata head can be achieved side download side play, that is, progressive download (http:// Baike.baidu.com/link?url= FTWQYBTQQR1BISYSCAKOQIYTBWOTFBYVFEMXEALSPRBNME6B5LWVLNZA-QGW6YGLFGBEPYBZQVUEB2TQQAEHBK), that's perfect, today on the Internet a search for Linux is good, Be excited to do a comparison test:

The first conclusion:

(1) In general, for text files, compression is more efficient than uncompressed transmission, but the effect is not obvious ( because the bottleneck is not the network transmission this block, but in compression, see test 1 below and 2,3 and 4 comparison);

(2) using edge packaging edge compression edge transmission edge decompression of the flow mode of transmission, transmission efficiency can be compared to direct scp/rsync way to improve 35%;

(3) specific to the flow of SSH and NC mode, because NC does not require user authentication, do not need to encrypt the transmission of data, the efficiency is slightly higher, the contrast effect is not obvious (because the bottleneck is not the network transmission this block, but in compression);

(4) in the actual use of more inclined to adopt the way of SSH, because: can use push or pull, and a command to do, the same source can have multiple concurrent, and NC needs to listen to the port on the receiving end, and then start the transmission at the end of the transfer, you need to execute 2 commands respectively. Worry: If the transmission of a third party at the same time to the receiver's listening port to send data, it is easy to cause data integrity, but the actual test found that the NC receiver can only connect with a sender to establish a connection for data transmission, if the data is being transmitted, then the third party to change the listening port data will not be transmitted, Only after the new listening port or the transmission is completed, and then re-enable the port for transmission, in short, still prefer the way ssh.

Test environment: centos5.5 Gigabit Local Area Network

Test catalog/var/log Size 8.9GB

[Email protected] ~]# du-h/var/log/
28K /var/log/prelink
8.0K /var/log/conman.old
8.0K /var/log/vbox
24K /var/log/cups
50M /var/log/redis
76K /var/log/nginx
6.1M /var/log/sa
8.0K /var/log/conman
8.0K /var/log/ppp
18M /var/log/audit
152K /var/log/php-fpm
8.8G /var/log/rabbitmq
12K /var/log/pm
16K /var/log/mail
8.9G /var/log/
[Email protected] ~]#

1, Direct Pure SCP copy time (5 ' 20 "):

[Email protected] ~]# time Scp-r/var/log/192.168.1.130:/root/test-dir/

Real 5m20.834s
User 3m29.049s
SYS 0m41.038s

2, first packaging compression and then transfer the time of decompression (3 ' 33 "+14" +1 ' 19 "= 5 ' 6"):
Pure compression Time:

[Email protected] ~]# time tar czf varlog.tar.gz/var/log
Tar:removing leading '/' from member names
Real 3m33.740s
User 3m28.068s
SYS 0m19.081s

Size after pure compression:

[Email protected] test-dir]# du-h. /varlog.tar.gz
399M .. /varlog.tar.gz

Time for pure transport of compressed packets:

[Email protected] ~]# time SCP varlog.tar.gz 192.168.1.130:~
[email protected] ' s password:
Varlog.tar.gz 100% 399MB 30.7mb/s 00:13

Real 0m14.024s
User 0m9.510s
SYS 0m1.283s

Time of Pure Decompression

[Email protected] ~]# time tar xzf varlog.tar.gz
Real 1m19.916s
User 0m49.498s
SYS 0m35.588s

3, the direct RYSNC does not enable the compression function transmission time (5 ' 12 "):

[Email protected] ~]# rsync-r/var/log/192.168.1.130:/root/test-dir
Rsync error:received SIGINT, SIGTERM, or SIGHUP (code) at RSYNC.C (260) [sender=2.6.8]
[Email protected] ~]# time Rsync-r/var/log/192.168.1.130:/root/test-dir
[email protected] ' s password:
Real 5m12.625s
User 3m55.503s
SYS 0m34.568s

4, direct rsync enable compression function transfer time (4 ' 36 "):

[Email protected] ~]# time Rsync-zr/var/log/192.168.1.130:/root/test-dir
Real 4m35.991s
User 4m40.208s
SYS 0m5.306s

5, side packaging edge compression edge transmission Edge decompression time (using SSH remote execution command push mode):

[Email protected] ~]# time tar czf-/var/log |ssh 192.168.1.130 tar xzf--c/root/test-dir/
Tar:removing leading '/' from member names
Real 3m33.711s
User 3m37.066s
SYS 0m22.210s

Edge Packaging edge compression edge transmission Edge decompression time (using SSH remote execution of the command pull mode):

[[Email protected] test-dir]# time ssh 192.168.1.131 tar czf-/var/log |tar xzf--c/root/test-dir/
Tar:removing leading '/' from member names
Real 3m33.772s
User 1m13.207s
SYS 0m55.302s

6, side packing edge compression edge transmission Edge Decompression time (using NC push method):

Receiver Listener Port 10086:

[Email protected] test-dir]# nc-l 10086 |tar xzf--c/root/test-dir/

The sending side begins the transfer:

[[Email protected] ~]# time tar czf-/var/log |nc 192.168.1.130 10086
Tar:removing leading '/' from member names
Real 3m31.218s
User 3m27.908s
SYS 0m15.839s

Edge Packing edge compression edge transmission Edge Decompression time (using NC Pull method):

This way doesn't seem to work!

Eof

Method and efficiency test of efficient synchronization of data--side packing edge compression edge transmission Edge decompression 20150105