Shell scripts implement scp upload (mutual CT interaction) and ftp upload, scpexpect

Shell scripts implement scp upload (mutual CT interaction) and ftp upload, scpexpect

There are two requirements recently. One is to copy files from each server to a server with a large disk on a regular basis every day, and copy the files on each server separately and delete the files on this server.

The second requirement is to transfer the backup files on servers with large disks through ftp. (Don't ask me why I need to bypass a layer of servers with large disks instead of directly transmitting data from each server over ftp, because ftp is too late. There is no second requirement for the first requirement)

I was originally engaged in java development, but the company does not have server O & M personnel, so I have to solve everything on my own. The scp command of shell script is studied. The initial solution is to separate shell script and scp interaction, and then call the scp interactive script in the main script, finally, I accidentally discovered that shell actually had the EOF function (Don't laugh, I basically didn't research it except for normal operations and queries in linux). I simply wrote a script and made a record.

The corresponding software must be installed for communications CT interaction. Tcl8.4.11-src expect5.45

Script 1:

Copy files from the specified directory to the specified directory on the other server at regular intervals. The local files are successfully deleted.

#!/bin/bash#owner huiyunfeiNOW_TIME=`date '+%Y-%m-%d %H:%M:%S'`echo "$NOW_TIME begin start backup.sh"FROM_DIR=/data/imageTAR_DIR=/data/tempLAST_DATE=$(date -d last-day +%Y%m%d)cd $FROM_DIRif [ ! -d "$FROM_DIR/$LAST_DATE" ]; then  echo "$NOW_TIME $LAST_DATE dir is not exist,begin exit"exit  fi  if [ ! -f "$TAR_DIR/$LAST_DATE.tar.gz" ]; thentar -zcf $TAR_DIR/$LAST_DATE.tar.gz $LAST_DATE/echo "tar $LAST_DATE.tar.gz success"elseecho "tar $LAST_DATE.tar.gz is exist"fiTO_HOST="172.19.75.191"USER="root"PASSWD="Shhz123654*"TO_DIR="/date/188/"/usr/expect/bin/expect <<-EOFset timeout -1spawn scp  -o "StrictHostKeyChecking no"  $TAR_DIR/$LAST_DATE.tar.gz $USER@$TO_HOST:$TO_DIRexpect "assword:"send "$PASSWD\n"expect "100%"expect eofEOFecho "scp $TO_HOST success"rm -rf $FROM_DIR/$LAST_DATErm -rf $TAR_DIR/$LAST_DATE.tar.gzecho "rm dir,dir.tar.gz success"echo "$NOW_TIME stop backup.sh"

The second script is described in detail.

#! /Bin/bash # owner huiyunfei #2011-1-24 # goal: # The local Folder/test1/test2/test3 will have a tar.gz # example: /test1/20180123.tar.gz/test2/20180123.tar.gz/test3/20180123.tar.gz # upload data to ftp data/test1 data/test2 data/test3 # generate the corresponding upload log token and continue uploading the next # after the general task is completed, delete the corresponding files in the local folder, exit the script # modify the following content in the script environment: # ftp_run_log total task log record # ftp_err_dir single folder Error Log Record (successful three times in a loop) Will be deleted) # local_file_dir local subfolder where the file to be uploaded is located # ftp_file_dir ftp subfolder # server_arr subfolder list # ftp_ip # ftp_user folder # ftp_run_log = '/home/yunfei/logs/ cron_ftp_run.log 'ftp _ err_dir = "/home/yunfei/logs/temperrorlogs/" echo "'date + % F" "% T ": "% n' Script: $0 start runing... ">>$ {ftp_run_log} t_last_day = 'date-d last-day + % Y % m % d' put _ file_name =" yesterday "# Yesterday's file 20180123ftp_ip = 116.62.175.185ftp _ User = yingjieftpftp_passwd = yingjiesftp ######################### return1send_file if the FTP automatic upload function has an error () {ftp-I-n $ {ftp_ip} 21 2 >3 3 <_ EOF _ user $ {ftp_user }$ {ftp_passwd} passive bin LCD $1 cd $2 put $ {put_file_name} bye_EOF _ # count the number of lines of the preceding FTP operation output error log records log_count = 'cat $3 | wc-L' [$ {log_count}-eq 0] & return 0 | | return 1 }########################## call the FTP upload function, return the status value required by until #120 seconds of sleep for each error, and the total cycle is three times. function run_sen D_file () {send_file $1 $2 $3 if [$? -Eq 0]; then echo "'date + % F" "% T": "% n' Send: $5 _ $ {put_file_name} to ftp_server was successfully. ">>$ {ftp_run_log} return 0 else echo" 'date + % F "" % T ":" % n' Send: $5 _ $ {put_file_name} more than $4 time. ">>1 {ftp_run_log} sleep 120 return 1 fi }########################### cyclically operate the handle () file in each folder () {local_file_dir = "/home/yunfei/data/$1/" ftp_file_dir = "/test/$1" ftp_err_log = "$ {ftp_err_dir} $1 _ $ {put_file _ Name} _ error. log "example if a folder does not contain a corresponding tar.gz file, directly perform the next folder if [! -D $ {local_file_dir}]; then echo "$ {local_file_dir} dir is not exist, begin next..." >>$ {ftp_run_log} return fi if [! -F $ {local_file_dir }$ {put_file_name}]; then echo "$ {put_file_name} file is not exist, begin next... ">>$ {ftp_run_log} return fi # loop 3 times if an error occurs, for a total of 6 minutes x = 1 I = 1 until [" $ I "-eq 0]; do [$ x-gt 3] & {echo "'date + % F" "% T": "% n' Send: $1 _ $ {put_file_name} to ftp_server was failed, Please check... ">>$ {ftp_run_log} exit} run_send_file $ {local_file_dir }$ {ftp_file_dir }$ {ftp_err_log} $ x $1 I = 'echo $? 'X = 'expr $ x + 1' done # Delete the temporary error log file [-f $ {ftp_err_log}] & rm $ {ftp_err_log} # Delete local data # rm-rf $ {local_file_dir }$ {put_file_name} server_arr = ("187" "188" "189 ") for server in $ {server_arr [@]}; do handle $ server; doneecho "'date + % F" "% T": "% n' Script: $0 done. ">>$ {ftp_run_log}