Automatically compare and delete scripts after backing up Files

Automatically compare and delete scripts after backing up Files

Recently, the company is working on a data backup plan, so I have compiled two backup documents, "scp automatic copy script" and "rsync + inotify for real-time file synchronization between servers, both the two solutions were successfully used. The leaders and colleagues both thought they could solve the backup problem. Then, they did not find any problems during the backup process, proving that the backup document I got was okay, the leaders also said that I did a good job.
However, during the next backup process, we found that the disk space on the server (that is, the server to be backed up) was full, some services on this server cannot be written into the server (this is an alarm email sent by monitoring disk space through nagios, but it is clearly set to trigger an alarm when the disk space exceeds 80% warning, if the number of crtical alarms exceeds 90%, but the disk space is full, the system receives the alarm. If you do not know where the alarm is set incorrectly, you are still searching.) So we plan to clear some files and obtain some space. When I log on to the server, I find that there are no files to be deleted, but the disk space on the server is full for more than five minutes, and the leaders are worried, in the end, there is no way to delete some files that have been backed up to the backup server on the server that is performing a backup task. This is the final solution, but the backup directory contains 8509 folders, each folder contains a txt file of around-. I was performing scp automatic copy. If you want to delete any files, you have to go to the backup end to check which files have been received by the backup end, then delete the file on the server. In this case, it is a waste of time and the workload is too large. Each file can be 3-5 MB. At the beginning, I cleaned up 20 files and got 83 MB, in this way, the application service on the server can write data to the server, but the file generation speed is too fast (in the past, when the disk space was large, it was useless to note), faster than the deletion speed, in this case, the disk will be full after a while. No way, I plan to use the script to solve this problem. After 30 minutes or so, I finally wrote a script and tested it several times on the test machine, this script was run on the production server. It was found that the deletion speed was very fast and the file was not deleted by mistake. This problem was finally solved. The following is a technical document for the O & M personnel manual I wrote for "automatic comparison and deletion of backup files.

Automatic comparison and deletion of backup files

I. Purpose

Because the application service in the engine server generates a folder every five seconds, each folder has a txt file of about to, the engine server's disk space is full and cannot work properly. To solve this problem, I have compiled two solutions: "ssh trust and scp automatic backup script" and "rsync + inotify Real-Time Synchronization of server files". However, I found that the backup task can be completed, it meets the requirements of the company's platform, but when backing up, it is very likely that the disk space of the engine server is full, and there is no way to work normally, or when the backup is complete, because of the large number of files, there is no way to know which are backed up and which are not backed up, so we cannot compare wc-l in time. Therefore, according to this situation, I wrote another "automatic backup file comparison and deletion script" to automatically compare and Delete backup files, solving the problem of full disk space, it saves a lot of time for our O & M work to solve other problems.

II. Implementation Mechanism

It is automated by means of scripts. This script is run on both the server and the backup end to generate logs, upload the logs on the backup end to the server end, and store the logs in the directory to be backed up on the server end, run the rm script to automatically compare and Delete backup files.

Iii. Script content explanationA.Server

#! /Bin/bashserver = "$ (ls-l/usr/local/lbs/file/area | awk '{print $9 }') "### view the directory to be backed up and copy the file name to the server_log log echo $ server>/tmp/server_log

Note: In the server script, server = "$ (ls-l/usr/local/lbs/file/area | awk '{print $9, ls-l/usr/local/lbs/file/area is the directory of your backup (the directory of the server to be backed up ), and the awk '{print $9}'. If you are using a server with an English font, use $9. If you are using a server with a Chinese font, use $8, you can use the cat/etc/sysconfig/i18n command to query the font of the current server. The query result is LANG = "en_US.UTF-8", which is an English font; the query result is LANG = "zh_CN.UTF-8", which is a Chinese font.

B.Back

#! /Bin/bashback = "$ (ls-l/home/beifen/area | awk '{print $9}')" ### view the backup directory, and the file name is responsible for echo $ back>/tmp/back_log in the back_log.

In the back script, ddir = "$ (ls-l/home/back/area20120208 | awk '{print $9, /home/back/area20120208 is the directory of your backup (the directory of the Backup Machine), and The awk '{print $9}' if you are on a server that uses an English font, use $9; if it is a Chinese font server, use $8,

You can use the cat/etc/sysconfig/i18n command to query the font of the current server. The query result is LANG = "en_US.UTF-8", which is an English font; the query result is LANG = "zh_CN.UTF-8", which is a Chinese font.

C.Server rm

#! /Bin/basha = "$ (cat/tmp/server_log)" # define a as the file name for viewing server logs. B = "$ (cat/tmp/back_log) "## define B as the file name forain $ B in the backup log; # If the file name on the server and the backup file name are consistent, delete the file name on the server, in this way, the file is no longer available on the server end, saving disk space. The file still exists on the backup end, and the backup task is scheduled. Dorm-rf $ bdone

4. Deployment script1At the backup end

Upload back to the backup end, grant the 764 permission, and run sh back. Then, a back_log log file is generated in/tmp, upload the file to the/tmp directory of the backup end;2, Server. upload the server to this server, grant 764 permissions, and run sh server. Then, a server_log file is generated in/tmp. B. upload rm to this server. The specific path is/usr/local/lbs/file/area, and the 764 permission is granted. C. run sh rm and wait for a while to automatically delete the files in the area (just delete the existing files in the Guizhou backup machine in the storage)5. Note: This script applies only to the backup end (/home/beifen/area) and backup end (/usr/local/lbs/file/area) between File Transfer and automatic deletion of files in the area, other situations are not applicable. If you want to back up files in other directories, you need to modify the content of the server and the back content of the backup end, upload rm to the directory on the server for backup.

The above is my O & M manual, which may be hard to understand in many places. I will perform some operations below to help you better understand.

The following figure shows the structure of the two taishi servers, including the host name, IP address, backup status (server or backup end), system, kernel, and number of digits;

1. Server (here I am savecenter)

Upload the server to the tmp/test directory of the server and grant the 744 permission. (This script needs to be modified according to the environment. My modifications are as follows)

#! /Bin/bashserver = "$ (ls-l/tmp/test | awk '{print $9}')" ### check the directory to be backed up, and copy the file name to the server_log log echo $ server>/tmp/server_log

2. Backup end (nagios here)

Upload back to the tmp/test directory of the server and grant the 744 permission. (This script needs to be modified according to the environment. My modifications are as follows)

#! /Bin/bashback = "$ (ls-l/tmp/test | awk '{print $9}')" ### check the backup directory, and the file name is responsible for echo $ back>/tmp/back_log in the back_log.

Now the script has been uploaded to the server, and the preparation is complete. Now we can test my script. But I will explain it first. The purpose of this experiment is to create 1-10 or 10 folders on the server (savecenter), and then on the backup end (nagios) create 1-5 or 5 folders. Use my scripts to check whether the folders can be automatically compared and deleted.

But why do we need to create 10 folders on the server and 5 folders on the backup end?

If the server backs up files to the backup end during backup, but the backup may not be completed and the backup task is still in progress, there may be 10 folders on the server, however, the backup end only receives five, but the server disk space is full. To obtain the space, you need to delete the files that have been backed up, therefore, I have created 10 folders on the server and five folders on the backup end, which indicates that the backup is not complete and is in progress. Then I use my script, delete the folders with the same name as the five folders on the backup end on the server. If they are correct, my script can automatically compare and delete the backup files.

3. Create 1-10 or 10 folders on the server. The result is as follows:

4. Create 1-5 folders on the backup end. The result is as follows:

5. Run the server script on the server now to obtain the server_log log, which contains the names of 1-10 and 10 folders under/tmp/test;

6. Run the back script on the backup end to obtain the back_log, which contains the names of 5 or 1 folders under/tmp/test.

7. Now upload the back_log of the backup end to the tmp directory of the server.

[root@nagiostmp]#scpback_log172.16.6.2:/tmpback_log100%100.0KB/s00:00

8. Upload rm to the tmp/test directory of the server and grant the 744 permission. (This script needs to be modified according to the environment. According to the current situation, my modifications are as follows)

#! /Bin/basha = "$ (cat/tmp/server_log)" # define a as the file name for viewing server logs. B = "$ (cat/tmp/back_log) "## define B as the file name forain $ B in the backup log; # If the file name on the server and the backup file name are consistent, delete the file name on the server, in this way, the file is no longer available on the server end, saving disk space. The file still exists on the backup end, and the backup task is scheduled. Dorm-rf $ bdone

9. Run the rm script on the server. It automatically compares server_log with back_log. If the name in server_log is in back_log, the content in the back_log is deleted, automatic comparison and Deletion

The following figure shows the running result of rm on the server.

As shown in the figure, the folders 1-5 and 5 have been deleted.

Below is

As we can see from the above, there are no changes in the 1-5 folders. Therefore, my "backup file automatic comparison and deletion script" Enables automatic comparison of Backup files and automatic deletion of tasks.

Summary: The "automatic comparison and deletion script for backup files" I wrote applies to the large number of backup files. A backup task is in progress, but you need to delete the files that have been backed up, or the backup task is completed, but there are many and complex backup files. The deletion needs to be compared and deleted, which is a waste of time and manpower, using this script and the "ssh trust and scp automatic copy script" and "rsync + inotify Real-Time Synchronization of server files" I wrote earlier can achieve automatic, real-time synchronization or backup of server files, when a backup task is in progress or the backup task is completed, the files that have been backed up are automatically compared and deleted, saving manpower and time.

Although I write a lot and it may be a little complicated, but the implementation is still very powerful, and the script is easy to understand. If you want to run it in your own environment, you need to modify the settings according to your needs. If you have any questions, please leave a message to me. I will help you answer the questions in a timely manner.

The following is the link between the original author's "ssh trust and scp automatic copy script" and "rsync + inotify Real-Time Synchronization of server files:

"Ssh trust and scp auto copy script" http://dl528888.blog.51cto.com/2382721/769519

"Rsync + inotify Real-Time sync server files" http://dl528888.blog.51cto.com/2382721/771533