Cross-network transmission of large files

Big file cross-network transmission skills O & m dba cross-network transmission of large files, such as the same as the kitchen in the toilet, the general good method is not obvious, and the advantages and disadvantages of the method may be very significant said: the price of the last mile of vegetables is doubled. Even if we often stop at 99.9% with thunder downloads, we can also see that the expensive file verification requires comprehensive consideration for the entire transmission, here, I only use compression to write and compress without file verification, which saves network bandwidth and improves transmission speed. resources such as CPU, memory, and I/O, it is best to use vmstat-n 5 to check whether I/O or CPU is the bottleneck. Don't worry about dogmatism, and compete for resources with the database Server. Dragging the DB Server down because compression will inevitably increase the CPU overhead! The following describes four methods: (1) rsync uses the binary difference algorithm of rsync to synchronize changed data only between networks. This is the advantage of rsync. If you know that most of the content in a file does not need to be transmitted, You can prioritize rsync.

[plain] [mysql@even ~]$ rsync -av /backup/ -e ssh root@192.168.1.110:/bak  root@192.168.1.110's password:   sending incremental file list  ./  cdio_bak.sql    sent 1061815 bytes  received 34 bytes  163361.38 bytes/sec  total size is 1061575  speedup is 1.00    [mysql@even ~]$ mv cdio_bak.sql.bz2 /backup  [mysql@even ~]$ rsync -av /backup/ -e ssh root@192.168.1.110:/bak  root@192.168.1.110's password:   sending incremental file list  ./  cdio_bak.sql.bz2    sent 308 bytes  received 34 bytes  76.00 bytes/sec  total size is 1061757  speedup is 3104.55  

 

Apparently, only the changed 308 bytes are synchronized this time. (Ii) scp: this is probably the simplest method, but the speed is slow because the disk needs to be read and written in every step.

[plain] [mysql@even ~]$ gzip -c cdio_bak.sql > cdio_bak.sql.gz  [mysql@even ~]$ scp cdio_bak.sql.gz root@192.168.1.110:/bak    [root@odd bak]# gunzip cdio_bak.sql.gz  

 

(3) pipelines are in place in one step, greatly reducing IO overhead. I usually use this type.

[plain] [mysql@even ~]$ gzip -c cdio_bak.sql | ssh root@192.168.1.110 "gunzip -c - > /bak/cdio.sql"  

 

(4) nc cross-network ssh is not the best. netcat enables data to "streaking" between networks to avoid system overhead caused by encryption. [Plain] ODD Machine [subject]: like a hungry pet dog, opening his mouth waiting for the little master to feed [root @ odd ~] # Nc-l-p 55555 | tar zxvf-EVEN machine [attack]: for example, a greedy Master, pour Dongdong into the dog's mouth [root @ even ~] # Tar zcvf-mm | NCS 192.168.1.110 55555