Linux shell script history file Cleanup script, by day, month, clean up the first n days of history files, delete designated size history files, historical file archive cleanup __linux

I don't know if there's anything to clean up. A directory history file needs to be cleaned. There will probably be a lot of other places to clean up the history files, which may be used.

I spent the past two days writing a script, cleaning more convenient, there will be a large number of historical documents to clean up the words can be used.

The commands used in scripts are only Linux, and are not supported like Solaris, so they can only be run on Linux. If it is NAS storage, it can be mounted on a Linux host to run script cleanup.

In addition, the script to find files with LS, but LS also has the maximum file limit (around 10w), if a lot of documents every day, every day is 10w+, run may be prompted too many files can not LS.

Also, there is a default restriction within the script that the file is created with the same time as the file name, and is not cleaned up if it is different, in order to avoid cleanup errors.

I have written two versions of the cleanup script, one is the clearfile.sh of direct cleanup files, the other is to archive files by day/month (archived into folders or compressed package form) Clearfile_ar.sh:

(i) pure clean up the document use clearfile.sh

The filename is a timestamp file, similar to the following (or other time format, described later)

Day/month cleanup : Modify the Clear_type parameter within the script, D is the day, M is the month

If it's a month, the script will be cleaned up in the month you enter and won't look at the time of day you entered.

To clean up all files in a specified period :

./clearfile.sh/wls/investop/tmp/20170515 20170516

to clean up fixed size files (ls-l display size) in a specified period:

./clearfile.sh/wls/investop/tmp/20170515 20170516 40

Other file date format cleanup :

The script default (trunc_date_flag parameter is N) is a file name that clears the format of the screenshot above, without any spacer characters.

If there are other spacers in the file name, such as *2017_05_16*.

First, you need to adjust the trunc () method in the script, the yellow date between the days of the separator to the required separator (by day and by the month two kinds of OH)

Then set the Trunc_date_flag parameter to Y to have the script clean up with the date format you specify

(ii) archival method of cleaning up historical documents, using CLEARFILE_AR.SH

Use the same way as clearfile.sh , the only difference is to set the del_flag parameters in the script.

Clear the same clearfile.sh by day/month

Clean up all files in the specified period with clearfile.sh

Clean up fixed size files with clearfile.sh during specified period

Other file date formats clean up with clearfile.sh

pattern 1:del_flag=n scripts file files by day/month into a time-named folder, such as

Mode 2:del_flag=y archive files into compressed file *tar.gz and delete archived folders

PS, for a large number of directories that need to be cleaned, you can write a very simple script to perform batch execution, similar

But to get rid of the confirmation in the script to perform this feedback, the script execution will still need you to enter Y.

The ~~~~~~~~~~~~~~~~~~ script is as follows ~~~~~~~~~~~~~~~~~~

clearfile.sh

#!/bin/bash ################################ # #参数设置 ################################ TODAY= ' Date +%Y%m%d ' DATE_BEGIN = ' Date +%y%m%d ' date_end= ' date +%y%m%d ' date_formation= '%y%m%d ' filesize= ' ' counting=0 #需要手工设置, monthly cleanup will not be cleaned according to the day of the input date,
Can only clean up the whole month.
Clear_type=m #需要手工设置, whether you need to convert the date format, if necessary, according to the format set TRUNC () method trunc_date_flag=n if [f count.tmp] then echo "" > Count.tmp Fi ################################ # #日期格式转换, Eg:trunc 20170512 ################################ trunc () {if ["D" = "$CL Ear_type "] then opr_date= ' echo $ | Sed-r ' s/^ (. { 4}) (. { 2}) (. { 2}) $/\1_\2_\3/g ' Else opr_date= ' echo $ | Sed-r ' s/^ (. { 4}) (. { 2}) $/\1_\2/g ' fi} ################################ # #按日清理递归 ################################ dayclear () {echo ===== =========================== "echo" Day clear working on $DATE _end "Doclear $DATE _end priday $DATE _end if [$DATE _en
 D-lt $DATE _begin] then echo "================================" echo "Total Cleard: $COUNTING" echo "Dayclear End" return fi Dayclear}

################################ # #按月清理递归 ################################ monthclear () {echo "==================== ============ "echo" clearfun working on $DATE _end "Doclear $DATE _end primonth $DATE _end If [$DATE _end-lt $DATE _be
 
 GIN] then echo "================================" echo "Total Cleard: $COUNTING" echo "Monthclear End" return fi 
 Monthclear} ################################ # #前一日日期, Eg:priday 20170421 ################################ priday () { Sec= ' date-d $ +%s ' pri_sec=$[${sec}-86400] pri_day= ' date-d @${pri_sec} +%y%m%d ' date_end= $PRI _day} ############# ################### # #前一月日期, Eg:primonth 201704 ################################ primonth () {#TMP_D = ' echo $ | cut-c 1- 6 ' #TMP_D =${tmp_d} ' tmp_d= ' echo $ ' "sec= ' date-d $TMP _d +%s ' pri_sec=$[${sec}-86400] pri_mon= ' date-d @${pri_s EC} +%y%m ' date_end= $PRI _mon} ################################ # #清理文件 ################################ doclear () {# Default date format, such as 20170512,201705 THe_date=$1 opr_date=$1 #转换OPR_DATE的日期格式, such as 2017-05-12,2017-05 if ["Y" = ${trunc_date_flag}]; Then trunc "clearing" if ["" = "${filesize}"] then #ROW = ' Ls-l *${opr_date}* | awk ' {print $9;output= ' ls-l ' $9;system (output);} ' | Wc-l ' Ls-ld--time-style +${date_formation} *${opr_date}* | GREP-V Archive | Awk-v date_str=${the_date} ' begin{count=0}{if (date_str==$6) {count+=1;output= ' rm ' $7;system (output);}}
  End{oo= "echo" Count "> Count.tmp"; system (OO);} ' Row= ' Cat count.tmp ' counting=$[$COUNTING + $ROW] Else #ROW = ' Ls-l *${opr_date}* | Awk-v The_size=${filesize} ' {if ($5==the_size) {output= ' ls-l ' $9;system (Output)}} ' | Wc-l ' Ls-ld--time-style +${date_formation} *${opr_date}* | Awk-v The_size=${filesize}-v Date_str=${the_date} ' begin{count=0}{if (the_size==$5 && date_str==$6) {count+=1 ; output= "rm" $7;system (output);}}
  End{oo= "echo" Count "> Count.tmp"; system (OO);} ' Row= ' Cat count.tmp ' counting=$[$COUNTING + $ROW] fi ECHo "Clear $THE _date finished, clear file $ROW"} ####################################### # main function # # to determine if the parameter is correct, you can 
Enter the following: # # Clean up All files: Clear 20150601 20161231 # clean fixed size byte file: Clear 20150601 20161231 40 # Clean fixed-size byte file: cleared 20150601 20161231 40 ####################################### #清理所有文件, if [$#-ne 3] && [$#-ne 4] then echo "Please enter the correct parameter" echo " such as cleaning the/nfsc/hipo/bak directory, all files from April 1, 2015 to December 31, 2016 (default by day cleaning): Clearfile/nfsc/hipo/bak 20150401 20161231 "echo" or cleanup/
	Nfsc/hipo/bak directory, from April 1, 2015 to December 31, 2016 size 40 bytes of files (default by day cleaning): Clearfile/nfsc/hipo/bak 20150401 20161231 "Exit 1 Else
	Date_begin=$2 date_end=$3 filesize=$4 date-d $2>&-err1=$?
	Date-d $2>&-err2=$? If [$ERR 1-eq 1] | | 	[$ERR 2-eq 1];
	 then echo "Date formation error, input the correct date string like 20170131." 	Exit 1 elif [$DATE _begin-gt $DATE _end];
	 Then echo "BEGIN date is later than end date, please correct." 	Exit 1 elif [$TODAY-lt $DATE _end]; Then Echo"End DATE is later than now date, please correct" elif [!-D $];
	 Then echo "DIR $ does not exist." Exit 0 Fi Fi cd $ Thedir= ' pwd ' echo ' >>>>clear the DIR $THEDIR ' echo ' >>>>clear file bet
 Ween $DATE _begin and $DATE _end. " if ["" = = "${filesize}"]; Then echo ">>>>every size of the files would be clear" else echo ">>>>clear filesize ${filesize} "fi if [" D "=" ${clear_type} "];  Then Echo ' >>>>clear set by day ' else Echo ' >>>>clear by MONTH ' fi read-p ' please confirm The information Above,enter Y to continue, and other to exit: "Anwser #确认继续 if [! -N "$ANWSER"] | | 
 [$ANWSER!= Y] then echo "Exit" Exit 0 fi echo "[' Date +%y%m%d-%h%m%s ']==start==" if ["D" = "$CLEAR _type"] Then echo "Day clear Begin" Date_formation= "%y%m%d" Dayclear else echo "MONTH clear Begin" Date_begin= ' echo $ | Cut-c 1-6 ' date_end= ' echo $ | Cut-c 1-6 ' date_formation= '%y%m ' monThclear fi echo "[' Date +%y%m%d-%h%m%s ']==end=="
 

clearfile_ar.sh

#!/bin/bash ################################ # #参数设置 ################################ TODAY= ' Date +%Y%m%d ' DATE_BEGIN = ' Date +%y%m%d ' date_end= ' date +%y%m%d ' date_formation= '%y%m%d ' filesize= ' ' counting=0 #需要手工设置, monthly cleanup will not be cleaned according to the day of the input date,
Can only clean up the whole month. Clear_type=m #需要手工设置, whether to delete archived files, only to keep the compressed file del_flag=n #需要手工设置, whether you need to convert the date format, if necessary, according to the format set TRUNC () method trunc_date_flag=n if [f COUNT.TMP] then echo "" > Count.tmp fi ################################ # #日期格式转换, Eg:trunc 20170512 ################# ############### trunc () {if ["D" = "$CLEAR _type"] then opr_date= ' echo $ | sed-r ' s/^ (. { 4}) (. { 2}) (. { 2}) $/\1_\2_\3/g ' Else opr_date= ' echo $ | Sed-r ' s/^ (. { 4}) (. { 2}) $/\1_\2/g ' fi} ################################ # #按日清理递归 ################################ dayclear () {echo =====  =========================== "echo" Day clear working on $DATE _end "Doclear $DATE _end priday $DATE _end if [$DATE _end -lt $DATE _begin] then echo ================================ "echo" Total Cleard: $COUNTING "echo" DaycLear end ' return fi dayclear} ################################ # #按月清理递归 ################################ Monthclea R () {echo "================================" echo "clearfun working on $DATE _end" Doclear $DATE _end primonth $DATE _e ND if [$DATE _end-lt $DATE _begin] then echo "================================" echo "Total Cleard: $COUNTING" E Cho "Monthclear end" return fi monthclear} ################################ # #前一日日期, Eg:priday 20170421 ########### ##################### Priday () {sec= ' date-d $ +%s ' pri_sec=$[${sec}-86400] pri_day= ' date-d @${PRI_SEC} +%Y%m%d ' d

ate_end= $PRI _day} ################################ # #前一月日期, Eg:primonth 201704 ################################ Primonth () {#TMP_D = ' echo $ | cut-c 1-6 ' #TMP_D =${tmp_d} ' tmp_d= ' echo $ ' "sec= ' date-d $TMP _d +%s ' pri_sec=$
 [${sec}-86400] Pri_mon= ' date-d @${pri_sec} +%y%m ' date_end= $PRI _mon} ################################ # #清理文件 ###################### ########## docleAr () {#默认的日期格式, such as 20170512,201705 the_date=$1 opr_date=$1 #转换OPR_DATE的日期格式, such as 2017-05-12,2017-05 if ["Y" = ${trunc_date _flag}]; Then trunc #创建归档文件夹 echo "clearing" if [!-D ${the_date} "archive"]; then mkdir "${the_date}" archive err=
  $? 	If [0!= ${err}]; Then echo "Failed to create archive dir under" ' pwd ' exit 0 fi echo "created the archive directory ${the_date}archi VE "chmod 777 ${the_date}" archive "Else Echo" the archive directory already exist "fi #如果要删除历史的 Archive file to determine if a history file has already existed in the compression pack if [f "${the_date}archive.tar.gz"] && ["Y" = "${del_flag}"]; then echo "File ${the _date}archive.tar.gz already exits, and again you choose to archive the file which had been, this'll archived the
  Old tar.gz file Tobe overwirted. "
  echo "You should confirm the date parameter you input, or you can mv the exits *.tar.gz file to other directory." Exit 0 fi if ["" = "${filesize}"] then ls-ld--time-style +${date_fOrmation} *${opr_date}* | GREP-V Archive | Awk-v date_str=${the_date} ' begin{count=0;} {if (date_str==$6) {count+=1;output= "mv" $ "" Date_str "archive/"; system (output);}}
  End{oo= "echo" Count "> Count.tmp"; system (OO);} ' If ["Y" = "${del_flag}"]; Then TAR-CF-${the_date} "archive/" |  gzip > ${the_date} "archive.tar.gz" Rm-rf ${the_date} "archive/" Fi row= ' cat count.tmp ' counting=$[$COUNTING + $ROW] Else ls-ld--time-style +${date_formation} *${opr_date}* | GREP-V Archive | Awk-v The_size=${filesize}-v date_str=${the_date} ' begin{count=0;} {if (the_size==$5 && date_str==$6) {count+=1;output= "mv" $ "" Date_str "archive/"; system (output);}}
  End{oo= "echo" Count "> Count.tmp"; system (OO);} ' If ["Y" = "${del_flag}"]; Then TAR-CF-${the_date} "archive/" |  gzip > ${the_date} "archive.tar.gz" Rm-rf ${the_date} "archive/" Fi row= ' cat count.tmp ' counting=$[$COUNTING + $ROW] fi echo "Clear $THE _date finished, clear file $ROW"} ####################################### # # Main Function # # to determine if the parameters are correct, you can enter: # # Clean up All files: Clear 20150601 20161231 # # Clean fixed size byte file: CLE Ar 20150601 20161231 40 # # Clean fixed size byte file: Clear 20150601 20161231 ####################################### #清理所有文件, if [$ #-ne 3] && [$#-ne 4] then echo "Please enter the correct parameter" echo ", such as cleaning/nfsc/hipo/bak directory, all files from April 1, 2015 to December 31, 2016 (default daily cleanup): C Learfile/nfsc/hipo/bak 20150401 20161231 "echo" or clean up the/nfsc/hipo/bak directory, File size 40 bytes from April 1, 2015 to December 31, 2016 (default daily cleanup): Clearfile/nfsc/hipo/bak 20150401 20161231 "Exit 1 else date_begin=$2 DA
	Te_end=$3 filesize=$4 date-d $2>&-err1=$?
	Date-d $2>&-err2=$? If [$ERR 1-eq 1] | | 	[$ERR 2-eq 1];
	 then echo "Date formation error, input the correct date string like 20170131." 	Exit 1 elif [$DATE _begin-gt $DATE _end];
	 Then echo "BEGIN date is later than end date, please correct." 	Exit 1 elif [$TODAY-lt $DATE _end]; Then echo "End DATE are later than now date, please correct" elif [! 	-D $];
	 Then echo "DIR $ does not exist." Exit 0 Fi Fi cd $ Thedir= ' pwd ' echo ' >>>>clear the DIR $THEDIR ' echo ' >>>>clear file bet
 Ween $DATE _begin and $DATE _end. " if ["" = = "${filesize}"]; Then echo ">>>>every size of the files would be clear" else echo ">>>>clear filesize ${filesize} "fi if [" D "=" ${clear_type} "];  Then Echo ' >>>>clear set by day ' else Echo ' >>>>clear by MONTH ' fi read-p ' please confirm The information Above,enter Y to continue, and other to exit: "Anwser #确认继续 if [! -N "$ANWSER"] | | 
 [$ANWSER!= Y] then echo "Exit" Exit 0 fi echo "[' Date +%y%m%d-%h%m%s ']==start==" if ["D" = "$CLEAR _type"] Then echo "Day clear Begin" Date_formation= "%y%m%d" Dayclear else echo "MONTH clear Begin" Date_begin= ' echo $ | Cut-c 1-6 ' date_end= ' echo $ |
 Cut-c 1-6 ' date_formation= '%y%m "Monthclear fi echo" [' DATE +%y%m%d-%h%m%s ']==end== ']

You are welcome to update your comments.

Thank you