linux技巧：一次刪除一百萬個檔案的最快方法

最初的測評

昨天，我看到一個非常有趣的刪除一個目錄下的海量檔案的方法。這個方法來自http://www.quora.com/How-can-someone-rapidly-delete-400-000-files裡的Zhenyu Lee。

他沒有使用find 或 xargs，他很有創意的利用了rsync的強大功能，使用rsync –delete將目標檔案夾以一個空檔案夾來替換。之後，我做了一個實驗來比較各種方法。讓我吃驚的是，Lee的方法要比其它的快的多。下面就是我的測評。

環境：

CPU: Intel(R) Core(TM)2 Duo CPU E8400 @ 3.00GHz

MEM: 4G

HD: ST3250318AS: 250G/7200RPM

Method

# Of Files

Deletion Time

rsync -a –delete empty/ s1/ 1000000 6m50.638s

find s2/ -type f -delete 1000000 87m38.826s

find s3/ -type f | xargs -L 100 rm 1000000 83m36.851s

find s4/ -type f | xargs -L 100 -P 100 rm 1000000 78m4.658s

rm -rf s5 1000000 80m33.434s

使用 –delete 和 –exclude，你可以選擇性刪除合格檔案。還有一點，當你需要保留這個目錄做其它用處時，這種方法是再適合不過了。

重新測評

幾天前，Keith-Winstein在回複Quora上的這個文章時說我之前的測評無法複製，因為操作的時間持續的太久。我澄清一下，這些資料過大，可能是因為我的電腦在過去的幾年裡做的事太多，測評中可能存在一些檔案系統錯誤。但我不確定是這些原因。現在好了，我弄了一天比較新的電腦，把測評再做一次。這次我使用/usr/bin/time，它能提供更詳細的資訊。下面就是新的結果。

(每次都是1000000個檔案。每個檔案的體積都是0。)

Command

Elapsed

System Time

%CPU

cs (Vol/Invol)

rsync -a –delete empty/ a 10.60 1.31 95 106/22

find b/ -type f -delete 28.51 14.46 52 14849/11

find c/ -type f | xargs -L 100 rm 41.69 20.60 54 37048/15074

find d/ -type f | xargs -L 100 -P 100 rm 34.32 27.82 89 929897/21720

rm -rf f 31.29 14.80 47 15134/11

原始輸出

# method 1~/test $ /usr/bin/time -v  rsync -a --delete empty/ a/        Command being timed: "rsync -a --delete empty/ a/"        User time (seconds): 1.31        System time (seconds): 10.60        Percent of CPU this job got: 95%        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:12.42        Average shared text size (kbytes): 0        Average unshared data size (kbytes): 0        Average stack size (kbytes): 0        Average total size (kbytes): 0        Maximum resident set size (kbytes): 0        Average resident set size (kbytes): 0        Major (requiring I/O) page faults: 0        Minor (reclaiming a frame) page faults: 24378        Voluntary context switches: 106        Involuntary context switches: 22        Swaps: 0        File system inputs: 0        File system outputs: 0        Socket messages sent: 0        Socket messages received: 0        Signals delivered: 0        Page size (bytes): 4096        Exit status: 0# method 2        Command being timed: "find b/ -type f -delete"        User time (seconds): 0.41        System time (seconds): 14.46        Percent of CPU this job got: 52%        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:28.51        Average shared text size (kbytes): 0        Average unshared data size (kbytes): 0        Average stack size (kbytes): 0        Average total size (kbytes): 0        Maximum resident set size (kbytes): 0        Average resident set size (kbytes): 0        Major (requiring I/O) page faults: 0        Minor (reclaiming a frame) page faults: 11749        Voluntary context switches: 14849        Involuntary context switches: 11        Swaps: 0        File system inputs: 0        File system outputs: 0        Socket messages sent: 0        Socket messages received: 0        Signals delivered: 0        Page size (bytes): 4096        Exit status: 0# method 3find c/ -type f | xargs -L 100 rm~/test $ /usr/bin/time -v ./delete.sh        Command being timed: "./delete.sh"        User time (seconds): 2.06        System time (seconds): 20.60        Percent of CPU this job got: 54%        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:41.69        Average shared text size (kbytes): 0        Average unshared data size (kbytes): 0        Average stack size (kbytes): 0        Average total size (kbytes): 0        Maximum resident set size (kbytes): 0        Average resident set size (kbytes): 0        Major (requiring I/O) page faults: 0        Minor (reclaiming a frame) page faults: 1764225        Voluntary context switches: 37048        Involuntary context switches: 15074        Swaps: 0        File system inputs: 0        File system outputs: 0        Socket messages sent: 0        Socket messages received: 0        Signals delivered: 0        Page size (bytes): 4096        Exit status: 0# method 4find d/ -type f | xargs -L 100 -P 100 rm~/test $ /usr/bin/time -v ./delete.sh        Command being timed: "./delete.sh"        User time (seconds): 2.86        System time (seconds): 27.82        Percent of CPU this job got: 89%        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:34.32        Average shared text size (kbytes): 0        Average unshared data size (kbytes): 0        Average stack size (kbytes): 0        Average total size (kbytes): 0        Maximum resident set size (kbytes): 0        Average resident set size (kbytes): 0        Major (requiring I/O) page faults: 0        Minor (reclaiming a frame) page faults: 1764278        Voluntary context switches: 929897        Involuntary context switches: 21720        Swaps: 0        File system inputs: 0        File system outputs: 0        Socket messages sent: 0        Socket messages received: 0        Signals delivered: 0        Page size (bytes): 4096        Exit status: 0# method 5~/test $ /usr/bin/time -v rm -rf f        Command being timed: "rm -rf f"        User time (seconds): 0.20        System time (seconds): 14.80        Percent of CPU this job got: 47%        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:31.29        Average shared text size (kbytes): 0        Average unshared data size (kbytes): 0        Average stack size (kbytes): 0        Average total size (kbytes): 0        Maximum resident set size (kbytes): 0        Average resident set size (kbytes): 0        Major (requiring I/O) page faults: 0        Minor (reclaiming a frame) page faults: 176        Voluntary context switches: 15134        Involuntary context switches: 11        Swaps: 0        File system inputs: 0        File system outputs: 0        Socket messages sent: 0        Socket messages received: 0        Signals delivered: 0        Page size (bytes): 4096        Exit status: 0

