Set up a way to automatically clean up files in a Linux system n days ago

Source: Internet
Author: User
Tags lstat rsync backup

The server produces a global backup file once a week, about 100G in size, and needs to be cleaned regularly.

Working time site access is large, server I/O high when the deletion of large data will have a bad impact on the server state. So I want to use the scheduled task to automate.

Under my backup directory/bakcup, each backup file names the directory names in the form of a date:

The code is as follows:

# ls

2013-12-23 2014-01-06 2014-01-20 2014-02-03

2013-12-30 2014-01-13 2014-01-27 2014-02-10

To delete a partial backup while preserving the part, you can use the Find command, such as I want to keep the files backed up in the last four weeks, every backup interval of seven days:

The code is as follows:

# find/bakcup/-maxdepth 1-type d-mtime +28

/bakcup/2014-01-06

/bakcup/2014-01-13

/bakcup/2013-12-23

/bakcup/2013-12-30

-maxdepth 1: Set lookup directory Depth of 1, only found in the/backup directory, if not with this parameter will be listed in the Subordinate directory files

-type d: Set Lookup type as directory

-mtime + 28: Find a 28 day ago directory

Use-exec parameter to connect Delete command after lookup ends

The code is as follows:

Rsync--delete-before-d/data/test/{};

So, the whole order is:

The code is as follows:

# find/bakcup/-maxdepth 1-type d-mtime +28-exec rsync--delete-before-d/data/test/{};

< p>

Finally you can put the command into the script, set crontab automatic execution.

Remind:

Before using the command, you should try the find part of the command on the server, and if you only find the directory you want to clean up, you can continue.

Do not rule out that some systems will find the./directory, be sure to see clearly, to prevent the occurrence of accidents.

In addition,-exec can be replaced with-OK, the effect is the same, before the deletion to remind users to confirm.

Comparison of efficiency between PS:RM command and rsync command

Rm

The RM command calls Lstat64 and unlink in large numbers, and you can speculate that a lstat operation was made from the file system before deleting each file.

The number of Lstat64 is lower than the total number of files, and there are other reasons, which will then be explained in another article.

Getdirentries64 This call is more critical.

Process: The first stage of the formal deletion work, the need for the GETDIRENTRIES64 call, batch read the directory (about 4K each time), in the memory of the establishment of RM file list; In the second phase, the Lstat64 determines the status of all files, and the third phase performs the actual deletion through unlink. There are more system calls and file system operations in these three phases.

Rsync

Rsync makes little system calls.

No Lstat and unlink operations are done for a single file.

In the early stages of command execution, Rsync opens a shared memory that loads directory information in a mmap way.

Directory synchronization only, do not need to do unlink for a single file.

In addition, in other people's evaluation, RM more context switching, will cause the system CPU consumption-for file system operation, simple increase the number of concurrent numbers does not always improve the operating speed.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.