Shell operation Typical case--ftp operation

Downloading files from an FTP server or uploading files to an FTP server is one of the more common scenarios in a production environment.

How the shell works with FTP is organized as follows:

Idea one: Using a shell to invoke clients such as FTP

FTP, through the shell call FTP and other clients, to complete the ftp file upload, download and other operations.

Mode One: FTP mode

A basic FTP tool that requires the installation of FTP (YUM-Y install FTP, after the same). A sample is as follows:

################################################### ##########
## Function Function: FTP upload, download multiple files, single file
## Example of use:
## DownloadFiles #Call directly without logging
## #Record the log and append the output to the log file
## You can stitch the date, batch number, etc. at the file name, directory name, etc. as needed.
################################################### ##########
IP = "" #FTP server address
PORT = "" #FTP port number
USER = "" #FTP username
PWD = "" #FTP user password
REMOTE_DIR = "" #Remote FTP server directory
LOC_DIR = "" #The corresponding local server directory
DEST_FILES = "" #Remote FTP server files

DownloadFiles () {
    ftp -ivn << EOF # -n is not affected by the .netrc file, which means that automatic login to the initial connection is prohibited (ftp defaults to read settings in the .netrc file)
                     # -i means turn off interactive prompts during multiple file transfers
                     # -v
    open $ IP $ PORT
    user $ USER $ PWD
    bin #bin is equivalent to bi, binary, etc., which means that it is transmitted in binary mode.
    #passive indicates FTP set to passive mode
    #prompt
    #Passive mode FTP is usually used when FTP clients behind the firewall access external FTP servers.
    #Active mode FTP means the server actively connects to the client's data port, and passive mode FTP means the server passively waits for the client to connect to its own data port
    #Passive mode FTP is usually used when FTP clients behind the firewall access external FTP servers.
    #Because in this case, the firewall is usually configured to not allow outsiders to access the host behind the firewall, and only allow connection requests initiated by the host behind the firewall to pass.
    #Therefore, in this case, FTP transmission in active mode cannot be used, while FTP in passive mode works well.
    cd $ REMOTE_DIR
    lcd $ LOC_DIR
    mget $ DEST_FILES #Download of multiple files
    #get $ DEST_FILE_NAME $ LOC_FILE_NAME #Single File Download
    close
    quit # or bye
    EOF
}

UploadFiles () {
    ftp -ivn << EOF
    open $ IP $ PORT
    user $ USER $ PWD
    bin
    #passive optional
    cd $ REMOTE_DIR
    lcd $ LOC_DIR
    mput $ DEST_FILES #Uploading multiple files
    #put $ LOC_FILE_NAME $ DEST_FILE_NAME #Single file upload
    close
    quit
EOF
} 

Way two: Sftp way

SFTP is an interactive file transfer program that runs and uses the same way as FTP commands, but the SFTP command uses SSH encryption for all information transmitted, and it also supports features such as public key authentication and compression for better security. The OpenSSH component must be installed using SFTP and it is dependent on the SSH service.

Way three: Lftp Way

Lftp is a powerful download tool that supports access to file protocols: FTP, FTPS, HTTP, https, hftp, fish. (where FTPs and HTTPS need to include the OpenSSL library when compiling). The Llftp interface is very much like a shell: command completion, history, allowing multiple background tasks to be performed and so on, is very convenient to use. It also has bookmarks, queues, mirroring, breakpoint continuation, multi-process download and other functions.

Need to install LFTP.

Examples are as follows:

(1) Use lftp alone

#! bin / bash
export NLS_LANG = american_america.AL32UTF8 #Set the character set

HOST = ""
USER = ""
PASSWD = ""
LOC_DIR = ""
DEST_DIR = ""

/ usr / sbin / lftp << EOF
     open ftp: // $ USER: [email protected] $ HOST
     mirror -R $ LOC_DIR $ DEST_DIR #Upload sync files
     mirror -vn $ DEST_DIR $ LOC_DIR #Download the file
     close
     exit
EOF

(2) lftp mixed with SFTP

#! / bin / bash

#yum -y install lftp openssh

lftp -u $ {USER}, $ {PWD} sftp: // $ {IP}: $ {PORT} << EOF #The port is optional and the default port of sftp is 22
cd [FTP server directory]
lcd [local directory]
mget / mput / get / put ..
bye
EOF

Note: Several FTP clients use the same way, you can use the Help command to see the use of assistance, and help command name Reference command parameters and other methods to assist in the completion of script writing.

IDEA two: Direct remote replication

According to the nature of the FTP service is file-based, so consider the same as the operation of files, direct remote replication, mobile

Mode four: Scp/rsync Way (1) SCP

Similar to SFTP, relies on SSH services to support encrypted transmissions. A bit: Security is good, the cost of less resources . Use format:

scp [parameters] [source path] [destination path] #The full format of the path: [email protected]: $ DIR

Both the local path and the remote path can be either source-side or target-side. Examples are as follows:

#! / usr / bin / expect
. ~ / .bash_profile
################################################### #################
## Call method:
## / usr / bin / expect This script path $ SRC_DIR $ USER ... (parameter list)
################################################### #################

set SRC_DIR [lindex $ argv 0]
set USER [lindex $ argv 1]
set HOST [lindex $ argv 2]
set src_file [lindex $ argv 3]
set PWD [lindex $ argv 4]
spawn scp -r $ SRC_DIR [email protected] $ HOST: $ DEST_DIR
  expect {
  "(yes / no)?"
    {
     send "yes \ n"
     expect "* assword:" {send "$ PWD \ n"}
  }

Note: 1. Use of the note *

scp -r $ LOC_DIR / * [email protected] $ DEST_HOST: $ DEST_DIR Copy all files in the folder (not including the folder itself)
scp -r $ LOC_DIR [email protected] $ DEST_HOST: $ DEST_DIR Copy all files in the folder (including the folder itself) 

2, if the target side of the file, directory, and the source end of the same name, will overwrite

3, when using SCP remote copy file, will be asked to enter a password, one way is to set up SSH password-free login. Otherwise, you need to install and use the expect tool to have the shell automatically enter the password.

(2) rsync

rsync (remote Synchronize) is a long-distance data synchronization tool that can synchronize multiple hosts with Lan/wan quickly, and Rsync uses the "Rsync algorithm" for synchronization between the local host and the remote host, which is not the whole transfer of the algorithm every time. It transmits only the different parts (incremental backups)of the data that are backed up between two computers, so it is very fast.

The advantages of rsync are as follows:
1, can be mirrored to save the entire directory tree and file system.
2, can easily do to maintain the original file permissions, time, soft links and so on.
3, without special Envoy permission to install.
4, has the optimized process, the file transfer efficiency is high.
5, can use rsh, ssh and other means to transfer files, of course, can also be directly connected through the socket.
6, support anonymous transmission.
In addition, the transfer speed is not a hierarchical level compared to the SCP. We often use rsync and SCP to transfer a large number of MySQL data in the LAN, found that rsync is at least 20 times times faster than SCP, so if you need to transfer huge amounts of data between Linux/unix servers, rsync is a very good choice. Of course, when there are a large number of small files, rsync results in a high disk IO load, while the SCP basically has a small impact.

How to use:

rsync [Optional parameters] source-side Target End

Reference: rsync reference: http://www.cnblogs.com/suihui/p/3799638.html

SCP Reference: http://www.cnblogs.com/peida/archive/2013/03/15/2960802.html

Idea three: Manipulate the URL of the FTP file address ftp file Address URL structure:

1) Full format: ftp://username:[email protected]:p ort/subdirectory or file
2) Shortcut format: ftp://[email protected]/subdirectory or file

Considerations when using the FTP URL address:

1) to prevent special characters in the URL, you can use the double quotation mark "URL"

2) FTP user's password, URL and other content may contain some special characters, when used, may error "-bash:!ifewbprint:event not Found", at this time, the content is enclosed in single quotation marks.

Description: Bash double quotes inside the exclamation point (!) will appear bash:!: Event not found-this error message however, it is not a problem to switch to single quotation marks . This is because bash has a strong single quote.

Quotation marks, which are called double quotes, are weak, and special characters are not translated in strong quotation marks, but are translated in weak quotes only. The exclamation point (!) happens to be a special character of Bash, which means a historical substitution, and

There's no command in the history command.

Reference: http://dusong1992.blog.163.com/blog/static/120930552201251710735240/

Way Five: Curl Way

Curl can easily upload and download ftp files under the shell, which is more advantageous than the FTP command, because it can download or upload an FTP file or even delete files under single command condition.

You need to install curl.

(1) List the files/directories on the FTP server:

curl ftp://malu.me/ --user name: passwd
curl ftp://malu.me/ --u name: passwd
curl ftp: // name: [email protected] #Simple writing 2 

(2) List only directory, do not show progress bar:

FTP://malu.me–u name:passwd-s

(3) Download individual files:

FTP://malu.me/size.zip–u name:passwd-o size.zip

(4) Upload a single file:

Curl–u Name:passwdFTP://malu.me/mp3/

(5) Delete files from the server (the DELE command that uses Curl to pass the FTP protocol)

Curl–u Name:passwdFTP://malu.me/-X ' DELE Mp3/size.mp3 '

(6) Download multiple files

Curl does not support recursive downloads, but you can download files in an array, such as we want to download 1-10.gif files that are continuously named:

Curl–u Name:passwdFTP://malu.me/img/[1-10].gif–o #O capital letter

To download multiple files consecutively:

Curl–u Name:passwdFTP://malu.me/img/[one,two,three].jpg–o #O capital letter

Reference: Upload ftp using Curl Download | Shacks Blog http://bolg.malu.me/html/2011/1239.html

Way Six: Wget way

wget is a command-line download tool, small but fully functional, it supports breakpoint download function, while supporting FTP and HTTP download method, support proxy server and easy to set up. Partial FTP functionality can be implemented accordingly. Need to install wget, more efficient than the FTP tool faster.

(1) Get a list of files

wget --ftp-user = $ USERNAME --ftp-password = $ PASSWORD url --no-remove-listing
wget --spider URL --no-remove-listing #The full format of ftpURL is used here: ftp: // $ USERNAME: [email protected] $ HOST / $ DEST_FILE_LIST 

#--no-remove-listing Save the listening file, get the HTML file of the FTP file list

(2) Download file

Example 1:

wget -r url -P $ LOC_DIR -nH --cut-dirs = n #url contains username and password
Add --cut-dirs = n to ignore n directories, which is a few levels from the root of the FTP server
Add -P dir to specify the local folder dir 

Example 2:

wget -nH -m --ftp-user = $ USER --ftp-password = $ PWD URL
-nH: Do not create a directory named after the host name.
--Cut-dirs: I want to remove the original number of directory layers, and start counting from the root directory. If you want to completely retain the original directory structure of FTP, do not add this parameter.
-m: Download all subdirectories and preserve the directory structure. 

Reference: http://www.jb51.net/LINUXjishu/86326.html

This article author original link: http://www.cnblogs.com/chinas/p/6957701.html, reprint please indicate the source!!!

Shell operation Typical case--ftp operation