Server date slicing file timed cleanup and compression backup shell script

This is a creation in Article, where the information may have evolved or changed.

With the server running, the business log is growing, before the article provided by the logger can be on the business log file by day, size, ordinary three mode segmentation, this script is to solve the server storage pressure, the daily split log files scheduled cleanup and compressed storage, need to use with crontab, The script is provided with the crontab script, which can be added by CRONTAB-E, the script adopts the system's own tar compression to zip format, the compression ratio is approximately the original file 1/4, the log file path needs to set the script file File_dir macro, File_dir_ Back for the backup file path, Del_file_num for the task cleanup from the current time of the file, Del_file_ret to keep the number of recent files, script run parameters are as follows:

Test for testing, resulting in the target format of the log files and compressed files

Clear for cleaning up files

Backup for backing up files

Run clean up files while backing up files

The crontab script is as follows:

* * * 1 sh/data/web/dsp-gateway/crontab.sh run >/data/web/dsp-gateway/crontab.txt

#!/bin/bash# @description Scheduled Tasks, based on the current system time, search forward del_file_num files, skip the most recent Del_file_ret files #* * 1 sh/data/web/dsp-gateway/ crontab.sh Run >/data/web/dsp-gateway/crontab.txt# @author chenbintao# @data2017 -07-1010:220.0.1 First Draft # 2017-07-1011:310.0.2 debugging through #2017-07-1014:410.0.3 to increase the retention of file compression and history of compressed file Cleanup #2017-07-1017:210.0.4 optimization cleanup process, first file cleanup, Clean up the original file by compressing the files one by one (TAR-ZCPF note the position of the P parameter) #2017 -07-1018:090.0.5 increase the running environment detection and increase the backup directory to prevent the backup files from being flume collected # 2017-07-1018:200.0.6 Add hints and version information #2017-07-1019:200.0.7 add run instructions to support cleanup and backup separation #2017-07-1021:010.0.8 test validation by # 2017-07-1021:310.0.9 Add Log Printing method # #VERSION =0.0.9# version del_file_num=31# number of deleted files (total number of deleted files) del_file_ret=7# Preserve file interval (how long the current time is not deleted) file_dir=/data/web/dsp-gateway/logs/#文件目录FILE_DIR_BACK =/data/web/dsp-gateway/logs/backup/#文件目录FILE_HEAD = ( Request response) #文件名称FILE_TAIL = (log) #文件后缀FILE_TAR =tar# compressed file suffix par_test=test# Test mode parameters par_clear=clear# cleanup file par_backup =backup# backup File par_run=run# official run # # #方法集function Ifo () {echo "======================================= file Cleanup/backup task script v${ VERSION} ($) ======================================= "echo"---0. Mode selection (Test: Generate Tests, run: Formal (cleanup & backup), clear: Clean files, Backup: Back up files) "echo"---1. Build tool Catalog "echo"---2. Clean up the history file "}function clear () {echo" 2. Clean the non-reserved raw/Compressed file "for ((i=${del_file_ret}+1;i<=${del_file_num};i++));d o# compute time file_date= ' date-d-${i}day +%f ' # Traverse the file header list for J in ${file_head[@]}; do# traversal file suffix list for k in ${file_tail[@]}; do# assembly file name File_name=${j}.${k}.${file_date} #删除原始文件cd ${file_dir}if [!-F "${file_dir}${file_name}"]; Thenprintln ' nonexistent file: ' ${file_dir}${file_name}elseprintln ' delete file: ' ${file_dir}${file_name}rm-rf ${file_dir}${file_name }fi# Delete compressed Files CD ${file_dir_back}tar_flie_name=${file_name}.${file_tar}if [!-F "${file_dir_back}${tar_flie_name}"]; Thenprintln ' nonexistent file: ' "${file_dir_back}${tar_flie_name}" elseprintln ' Delete file: ' "${file_dir_back}${tar_flie_name}" Rm-rf "${file_dir_back}${tar_flie_name}" fidonedonedone}function backup () {echo "3. Compress the retention file and delete the original file" for ((i=1;i<=${del_ file_ret};i++)); do# calculation time file_date= ' date-d-${i}day +%f ' #遍历文件头列表for j in ${file_head[@]}; do# traversal file suffix list for k in ${file_tail[@]}; do# assembling the compressed file name File_name=${j}.${fIle_tail}.${file_date}if [!-F "${file_dir}${file_name}"]; thencontinuefi# Generate compressed Files TAR_FLIE_NAME=${FILE_NAME}.${FILE_TAR}CD ${file_dir_back}if [!-F "${file_dir_back}${tar_flie _name} "]; Thenprintln ' compressed file: ' ${file_dir_back}${tar_flie_name} ' TAR-ZCPF ' ${file_dir_back}${tar_flie_name} ' ${FILE_DIR}${ File_name}elseprintln ' already exists file: ' ${file_dir_back}${tar_flie_name} ' fi# clean original file Rm-rf ${file_dir}${file_name}println ' Delete the file: "${file_dir}${file_name}donedonedone}function Test () {echo" 1. Generate the testing file "CD ${file_dir}for (I=1;i<=${del_file_ num};i++)); do# calculation time file_date= ' date-d-${i}day +%f ' #遍历文件头列表for j in ${file_head[@]}; do# traversal file suffix list for k in ${file_tail[@]}; do# assembly file name CD ${file_dir}file_name=${j}.${k}.${file_date} #创建原始文件touch ${file_dir}${file_name} #创建压缩文件tar_flie_name =$ {FILE_NAME}.${FILE_TAR}CD ${FILE_DIR_BACK}TAR-ZCPF "${file_dir_back}${tar_flie_name}" ${FILE_DIR}${file_name}# println "Test: Create File ${file_dir}${file_name} ${file_dir_back}${tar_flie_name}" Donedonedone}function println () {echo ' Date ' +%y-%m-%d%h:%m:%s "": "$*}############################################################################################## Run code echo "0. Pre-run Prep" CD ${FILE_DIR}IF [!-d "${file_dir}"]; Thenecho "Log directory does not exist, quit! "Exitfiif [!] -D "${file_dir_back}"]; Thenecho "Backup directory does not exist, create! "Mkdir-p ${file_dir_back}fiifoif [$#-gt 0];thenif [= = ${par_test}];thentestexitelif [= = ${par_run}];thenclea rbackupexitelif [= = ${par_clear}];thenclearexitelif [= = ${par_backup}];thenbackupexitfiecho "parameter error, exit!" Exitfiexit

261 Reads