Delete a large number of files in Linux

Mainly refer to the http://www.slashroot.in/which-is-the-fastest-method-to-delete-files-in-linux

Create 0.5 million files first

➜  test    i  $(  );  text >>$i.txt;                 

1. rm

➜  test    -f * /home/hungerr/test [yn]?:  -f *  .63s user .29s system % cpu  total

Rm does not work because there are too many files.

2. find

➜  test    ./ -type f -exec  ./ -type f -exec  {} \;  .86s user .13s system % cpu : total

About 43 minutes, my computer ...... While watching the video, you can delete it.

3. find with delete

➜  test    ./ -type f - ./ -type f -delete  .43s user .21s system % cpu : total

It takes 9 minutes.

4. rsync

First, create an empty folder named "blanktest ".

➜  ~   rsync -a --delete blanktest/ test/-a --delete blanktest/ test/  .59s user .86s system % cpu  total

16 s, very good and powerful.

5. Python

import osimport timestime=time.time()for pathname,dirnames,filenames in os.walk('/home/username/test'):    for filename in filenames:        file=os.path.join(pathname,filename)        os.remove(file)ftime=time.time()print ftime-stime      

➜  ~

It takes about 8 minutes.

6. Perl

➜  test    -e  -e   .28s user .23s system % cpu  total

16 s, this should be the fastest.

 

Statistics:

Command	Time-consuming
Rm	Too many files, not available
Find with-exec	0.5 million it takes 43 minutes for a file
Find with-delete	9 minutes
Perl	16 s
Python	8 minutes
Rsync with-delete	16 s