Operation Record of fast migrating massive files under Linux

There is such a migration of large files operation and maintenance scenario : For example, because the existing Web server configuration is not enough, you need to do a site migration (is to migrate to another high-configuration server with), the site directory has a large amount of small files, about 100G , picture files are mostly. It takes a few days to visually copy the past. So the question is, how do you migrate your website data in this situation? In addition, the site is still in operation, the daytime is flatly can not stop, only run late at night to stop a few hours .

1. Synchronize with Rsync. This method is slow, but fortunately for continuous transmission, in the case of low bandwidth or instability of the site, it is strongly recommended to use this method:
1) First modify the old station upload image function, to ensure that the newly uploaded image is saved to another new directory address;
2) Use Rsync to synchronize the old pictures to the new machine; rsync can speed up the operation without affecting the external services of the website.
3) can write a rsync synchronization script, timed execution at night (such as 1 o'clock in the morning execution, 6 o'clock end sync), 100G files, not a few nights can be done.
4) to the old station pictures are all synchronized, and then once the new uploaded pictures rsync synchronization past. and migrate the site code.
2. If the Internet is fast and the network is stable, you can consider the transfer of tar packaging (compression) . However, after packing, in a stop cycle to complete the migration, for 100G of the amount of file transfer, this method is not very reliable.
3. can be chunked packaged, For example, according to the image size appropriate block filter (find) packaging, and then transfer.
4. If the data over HTTP (wget).
5. Take down the hard drive of the old station server directly,

Example Description:
For example, the native site directory/var/www/html has more than 1 million small files, you need to migrate these files to the remote server 192.168.1.101/var/www/html directory.
Operation Idea:
Directly with rsync to the file one of the migration past, because the number of files is relatively large, if a sudden operation in the loop script, will be very slow.
So decided to use a batch operation, using piecemeal method .

In order to test the effect, you can first build a number in the/var/www/html directory
[Email protected] ~]# cd/var/www/html
[[email protected] ~]# for i in ' seq 1 1000000 ';d o touch test$i;done

1) using the Rsync synchronization method

[Email protected] ~]# cat/root/rsync.sh

#!/bin/bashhome=/var/www/htmlcd $home  If [' pwd ' = = $home];then     a= "1 100000 200000 300000 400000 500000 600000 70 0000 800000 900000 "             //lower than 1 million units, that is, 100,000 units for B in the     $a do       c= ' expr $b + 10000 '                                       //than 100,000 lower one unit       for Loop in $ (ls-l|sed-n "$b, $c" p|awk-f "" ' {print $9} ')       does         rsync $loop 192.168.1.101: $home       done    Done
   fi

[Email protected] ~]# chmod 755/root/rsync.sh
[Email protected] ~]#/bin/bash/root/rsync.sh

2) Adopt wget mode (assuming native IP is 192.168.1.99)

[[email protected] ~]# CAT/ROOT/RSYNC.SH#!/BIN/BASHHOME=/VAR/WWW/HTMLCD $home  if [' pwd ' = = $home];then     a= "1 10 0000 200000 300000 400000 500000 600000 700000 800000 900000 "for                 b in $a     does       c= ' expr $b + 10000 ' for                                              loop In $ (ls-l|sed-n "$b, $c" p|awk-f "" ' {print $9} ')       do         /usr/bin/ssh [email protected] "/usr/bin/wget http://192.168 .1.99/$loop-P/var/www/html "Done"  fi

Operation Record of fast migrating massive files under Linux