Efficiency of the copy and delete operations on a large number of files

When copying a large number of files, it is inconvenient to use CP (for example, after the command is accidentally interrupted, it cannot be resumed; if it is forcibly overwritten, it will waste time; if it is not overwritten, echo "N" | CP-r/source/destination can be used to complete the task. However, after my experiment, the overwriting prompt passed to the CP will also delay time, so there is also a waste of time for existing files)

Rsync has many features to avoid the inconvenience of CP mentioned above, but considering the large number of files, you have to consider efficiency.

The following section compares the efficiency of copying and deleting a large number of files based on CP, Rm-RF, and rsync.

The experiment copies 11041 GB of data (the size of the files is usually less than 25 MB)

I. Copy

1. CP

Time CP-RP-V/APP/glusterfs/networkdisk/home/

Real 16m29. 876 S
User 0m0. 889 s
Sys 1m59. 575 s

2. Rsync Mode

Time rsync-Av/APP/glusterfs/networkdisk/home/

Real 16m34. 557 s
User 8m53. 267 s
Sys 4m56. 069 s

Ii. Delete

1. Use Rm-RF

Time Rm-RF/home/networkdisk

Real 0m3. 341 s
User 0m0. 002 s
Sys 0m3. 297 s

2. Use rsync -- delete

Rsync-A -- delete/empty/home/networkdisk

Real 0m1. 709 s
User 0m0. 014 s
Sys 0m1. 573 s

Through the comparison above:

The replication speed seems to be the same as the replication speed.

In the delete operation, we can see that rsync-delete is about 2 s faster than RM-RF.

In order to get more credible results, I made a comparison of the operations on 10 million small files, as shown below:

Generate 10 million small files

For I in {1 .. 100000}; DO dd If =/dev/Zero of =/APP/test/$ I BS = 100 K; done

I. Copy

1. Time CP-RP-V/APP/test/home/

Real 6m21. 137 s
User 0m1. 848 S
Sys 0m34. 099 s

2. Rsync-Av/APP/test/home /,

Real 3m56. 437 s
User 1m53. 502 s
Sys 1m21. 125 s

Ii. Delete

1. rm-RF/APP/test

Real 0m4. 527 s
User 0m0. 143 S
Sys 0m4. 339 s

2. Rsync-A -- delete/empty/APP/test

Real 0m0. 122 S
User 0m0. 000 s
Sys 0m0. 009 s

Through the above operations on 10 million small files, we can conclude that for copying and deleting a large number of files, rsync has great advantages over CP and RM.

The last article is attached.

Http://www.linuxeden.com/html/news/20130613/140243.html