Linux Polling directory FTP transfer files

Before the company, on the Linux server need to write a shell script, the function is as follows: Scheduled task 5 Seconds to execute, polling the current machine (127.0.0.1) a directory, and a directory of all qrytyp* The file at the beginning is transferred to the B directory of the other machine (10.32.64.128), and the filename is qrytyp*.

Here are a few questions to consider: There is now a file QRYTYP123456 that needs to be transferred,

The 1.qrytyp123456 reaches the a directory, but the file is too large and still in transit. And just by the scheduled task polling, so the target file under B directory will be incomplete.

2. Assuming that the QRYTYP123456 has been transferred, there is a scheduled task polling the file, and in the FTP transfer process, just again by the second scheduled task polling to this file, resulting in a file and two scheduled tasks to transfer, not what we want to see the results.

3. How to ensure the FTP transmission is successful during the multi-tasking transfer of large files?

The whole idea of solution is as follows:

1. Use the lsof command to see if the file is occupied by another process

2. If the file is not occupied, indicating that the transfer is complete, the file QRYTYP123456 to QRYTYPE123456.TMP,FTP transmission is the qrytyp123456.tmp file to target machine B's temporary directory (TMP), After the transfer is complete, use the Rename directive to duplicate the file and move it to the target directory.

3. In order to transfer the same file for multiple tasks, all the qrytyp* file names polled by each scheduled task are written to the file.lst.${pid_number} file, pid_number = [process number] [current time], so that each task processes the polling file. does not cause concurrent transmissions.

4. According to the FTP transfer log, can be based on the log "226 Transfer complete" (the specific log will also need to be based on the version of Linux), but generally has been "226" beginning, so the log appears "226 ..." means that the file transfer is completed.

Said so much, on the most practical code:

The entire module script is placed in the Tools directory, respectively: Dgyc_crontab_second_5.sh,dgyc_main_for_cronb_second_5.sh,file_ftp2dgyc_for_crontab_ Second_5.sh, there are a few points to note:

1.dgyc_crontab_second_5.sh This file can be executed by CRONTAB-E, adding a line: * * * * * * dgyc_crontab_second_5.sh, save exit.

Here is polling every 5 seconds, you can modify the Interval_second, shorten the polling time, preferably can be divisible by 60, this will be more accurate.

2.dgyc_main_for_cronb_second_5.sh:loc_snd_dir=/si/usr/dgyc/fsend is the directory you want to poll, Loc_snd_tmp=/si/usr/dgyc/tmp is used to put the log.

Set the filter rule file_name_filter_regulation= "qrytyp*", if the LS qrytyp* this wildcard form does not meet your filtering requirements, you can filter by the pipe "|", and then using awk or grep regular expressions can also be filtered.

Rmt_snd_dir,rmt_tmp_dir is the directory of the target machine, generally use "/", because the target machine FTP is based on the directory to open.

3.file_ftp2dgyc_for_crontab_second_5.sh: To modify the target machine address, user name, password.

　　Special Reminders:

1. Copy the following code into a text editor, the line break default is "CRLF", but Linux is "LF" end, need to change.

2. In the process of file execution, see if *.sh has "x" permission, Novice will often make this mistake Ah!!!

#!/bin/ksh#if the script startuped by Crontab,#It'll ececute dgyc_main_for_cronb_second_5.sh per 5 seconeds.base_home=/Toolsinterval_second=5Let interval_count=60/Interval_secondscript_file=$BASE _home/dgyc_main_for_cronb_second_5.sh for((i=1;i<=${interval_count}; i++)); Do  ${script_file}2>/dev/null &Sleep${interval_second} Done

#!/bin/ksh##file directory#script_dir=/Tools#update1loc_snd_dir=/si/usr/dgyc/Fsendloc_tmp_dir=/si/usr/dgyc/Tmprmt_snd_dir=/Rmt_tmp_dir=/Current_date_time=$ (date"+%y%m%d%h%m%s") Pid_number=$$$CURRENT _date_timeTmp_file_suffix=". tmp"#----------------------------------------------------------------##Send the Txn file##----------------------------------------------------------------##get the Txn file list#Cd$LOC _snd_dirRM- F $LOC _tmp_dir/file.lst.$PID _number>/dev/null 2>&1ls-1- Fqrytyp* | grep-v [/$] | Grep-v ["$TMP _file_suffix"$] | whileRead Line Do/usr/sbin/lsof |grep$LINE|grep-v lsof|grep-v grep >/dev/null 2>&1if["$?"="1"] then MV$LINE $LINE $tmp_file_suffixEcho$LINE>>$LOC _tmp_dir/file.lst.$PID _number            #crontab Task would send a file only.             Breakfi Doneif[ !- F "$LOC _tmp_dir/file.lst. $PID _number"] Then exit1fifile_size= ' Ls-l$LOC _tmp_dir/file.lst.$PID _number| awk ' {print $} 'if[$FILE _size -eq0]then RM- F $LOC _tmp_dir/file.lst.$PID _number>/dev/null 2>&1fi##get the file name and put#if["$FILE _size"!="0"]then Cat$LOC _tmp_dir/file.lst.$PID _number| whileRead Line Do        #Update3/usr/sbin/lsof |grep$LINE $tmp_file_suffix|grep-v lsof|grep-v grep >/dev/null 2>&1if["$?"="1"] Then$SCRIPT _dir/file_ftp2dgyc_for_crontab_second_5.sh$LOC _snd_dir $LOC _tmp_dir $LINE $RMT _snd_dir $RMT _tmp_dir $PID _number $TMP _file_suffixfi donefiif[- F "$LOC _tmp_dir/file.lst. $PID _number"]then RM$LOC _tmp_dir/file.lst.$PID _numberfi#****************************************************************##End of File##****************************************************************

#!/bin/ksh##FTP constant Variable#Ftp_addr="10.32.64.128"Ftp_user=dgycftp_pwd=dgyc1113##input Variable#Loc_snd_dir= $Loc_tmp_dir= $file_name= $Rmt_snd_dir=$4Rmt_tmp_dir= $Pid_number=$6Tmp_file_suffix=$7ftp_success_msg="^226"Ftp_information_log=$LOC _tmp_dir/"Ftp_information_"$PID _number". Log"##Check the file#Cd$LOC _snd_dirls-L$FILE _name$tmp_file_suffix>/dev/null 2>&1if["$?"="1"]then Exit1fi##Copy the orignial file to a new file#With ". tmp" in the name.#Tmp_file_name=$FILE _name$tmp_file_suffix##put the file#Ftp-inv$FTP _addr<<! >>$FTP _information_logUser$FTP _user $FTP _pwdBINCD$RMT _tmp_dirput$TMP _file_namels-Lrename$RMT _tmp_dir/$TMP _file_name $RMT _snd_dir/$FILE _nameClosequit!ifGrep"$FTP _success_msg" $FTP _information_log; Then#If FTP transfer success,it would remove temp fileRm$TMP _file_nameElse    #If FTP transfer failure,it would rename temp file to fileMv$TMP _file_name $FILE _namefiif[- F $FTP _information_log]then RM-rf$FTP _information_logfi#****************************************************************##End of File##****************************************************************

Linux Polling directory FTP transfer files