Linux File Transfer (2)

Curl is another tool that can be used for file transmission. it is packaged as a command line tool for the libcurl Library. The libcurl Library provides APIs for relevant functions and can be called in programs. The use of libcurl Library is beyond the scope of this article ....

Curl
Another tool that can be used for file transfer is curl, which is a command line tool packaging for the libcurl Library. The libcurl Library provides APIs for relevant functions and can be called in programs. The use of libcurl Library is beyond the scope of this article. Curl uses the URL syntax to transmit files. it supports FTP, FTPS, HTTP, HTTPS, TFTP, SFTP, TELNET, and other protocols. Curl is powerful and provides many features including proxy support, user authentication, FTP upload, HTTP post, SSL connection, and file resume.
Basic syntax
Curl [options... ]
There are more than 80 download parameters, and each function of curl relies entirely on these parameters. The following examples illustrate some basic usage of curl.
Instance used:
Obtain the GNU homepage
Curl http://www.gnu.org
Obtain the README file in the root directory of the gnu ftp server.
Curl ftp://ftp.gnu.org/README
Get the curl definition from a dictionary
Curl dict: // dict.org/m:curl
You can specify the user name and password in the url or use the-u parameter.
Curl ftp: // name: passwd@machine.domain: port/full/path/to/file
Curl-u name: passwd ftp://machine.domain: port/full/path/to/file
Curl prints the content obtained from the specified url to the standard output. If you want to save the file to a local file, you can use-o or the-O parameter to specify the file name on the remote host (if the file name is not provided in the url, this operation will fail)
Http://www.gnu.org curl-o gnu.html
Curl-O http://www.gnu.org/index.html
Use the-x option to connect using a proxy
Curl-x my-proxy: port ftp://ftp.somesite.com/README
Upload by using the-T option of curl
Curl-T-ftp://ftp.upload.com/upfile
This command reads data from the standard input and uploads it to the upfile file on the remote FTP server. You can also upload a local file.
Curl-T localfile-a ftp://ftp.upload.com/upfile
The-a parameter indicates that the content in localfile is appended to the end of upfile.
In general, curl is suitable for automatic file transmission or operation sequence. it is a good tool to simulate users' behaviors on Web browsers. Libcurl is a good choice, especially when it needs to be called in a program.
Rsync
Rsync is an efficient remote data backup and Image mirroring tool that can quickly synchronize files between multiple hosts. it has the following features:
Supports copying link, owner, group information, and permission information;
Transmission through remote shell (ssh, rsh;
Installation and Use without special permissions;
Streamlined file transmission mode, high file transmission efficiency;
Supports Anonymous operations;
It should be mentioned that rsync has a superior performance advantage compared with several other Linux file transmission methods, and its file synchronization speed is quite fast, thanks mainly to the transmission algorithm used by rsync. In short, The rsync algorithm can calculate the data to be backed up in a short period of time, and only transmit the differences between the source file and the target file, thus reducing the amount of data transmitted over the network, this allows you to quickly back up images. The following describes the basic principles of the rsync algorithm through A typical application scenario: both host A and host B copy the same file, and the user updates the copy on host, host B synchronizes the updated files through the rsync algorithm. The following are the steps to implement the algorithm:
Host B divides the original copy into N non-overlapped blocks (the part size at the end of the file may be less than N), and verifies these data blocks in two different ways: 32-bit rolling weak verification and 128-bit MD4 strong verification. Weak verification is faster than strong verification.
Host B sends the weak and strong verification results of each data block to host.
Host A performs A weak check on each data block whose length is N in the updated file copy and matches the weak check value received from host B. If the value is the same, the strong check matches. Due to the rolling feature of weak verification, you can quickly filter data blocks that need to be synchronized. The calculation workload of this algorithm is mainly concentrated on host.
Through the above calculation, host A Sends different parts of the file to B, and host B receives the differences between the two copies to synchronize the updated files.
Through the above method, rsync avoids the transmission of the same data and reduces the waste of network bandwidth. In terms of time, a round trip is required throughout the process, which ensures the superior performance of rsync to some extent.
Users can download the latest version of rsync installation from the official website http://rsync.samba.org. During use, you need to install rsync on the server and the client respectively. the server and the client use the same rsync package to achieve remote mirroring and regular synchronization updates. It must be noted that an rsync server can back up data from multiple clients at the same time; multiple servers can back up data from one client. The default rsync port is 873. the server receives anonymous or authenticated backup requests from the customer on this port.
Before using rsync, you need to configure the server. the configuration file is/etc/rsyncd. conf for authentication, access, logging, and other control. The configuration file includes global parameters and module parameter settings. In the rsyncd. conf file, all parameters before [module] are global parameters. you can also define the module parameters in the global parameters section. in this case, the value of this parameter is the default value of all modules. Global parameter setting specifies the port number used by the program, specify the message file, log file pid file, and the level at which the log message is sent. Module parameters mainly define which directory on the server needs to be synchronized. You can specify multiple modules according to different requirements. each module corresponds to a directory tree to be backed up. that is, if there are N directory trees to be backed up, N modules need to be matched. Many parameters can be defined in the module. common parameters are as follows.
 
Table 4. list of common module parameters in the rsyncd. conf configuration file

 

Option	Description
Comment	Module information description. this description, together with the module name, is displayed to the customer when the customer connects to the module list. No description is defined by default.
Path	Specifies the directory path for backup. this parameter must be specified.
Max connections.	Specify the maximum number of concurrent connections to protect the server. connection requests that exceed the limit will be notified and then try again. The default value is 0.
Log file	Specify log files
Read only	Set whether to allow users to upload files. If the value is true, any Upload request will fail. if the value is false and the client has the server directory read/write permission, the upload can be performed. The default value is true.
Write only	Set whether to allow users to download files. If it is true, any download request will fail. the default value is false.
List	Sets whether to list modules that can be used by the customer upon request. If this parameter is set to false, a hidden module is created. The default value is true.
Fake super	Allow all permissions on files, without the need for background services to perform operations with root permissions
Filter	Sets the filter list to determine which files can be accessed by the client.
Hosts allow	Specifies the IP address that allows the customer to connect. It can be a single IP address or the entire network segment. Multiple IP addresses or CIDR blocks must be separated by spaces. By default, all hosts are allowed to connect.
Dont compress	Specifies the files that can be transmitted without compression. the default value is *. gz *. tgz *. zip *. z *. rpm *. deb *. iso *. bz2 *. tbz.
Pre-xfer exec, post-xfer exec	Set the command that can be executed before/after file transfer. If the command executed before File transfer fails, cancel the transfer operation.

Basic syntax

Remote shell:

Rsync [OPTION] [USER @] HOST: SRC DEST

The remote shell (such as ssh and rsh) is used to copy the contents of the remote machine to the local machine. The SRC address path is separated by a single colon.

Rsync [OPTION] SRC [USER @] HOST: DEST

Use remote shell (such as rsh and ssh) to copy contents from local machines to remote machines. The DEST address path is separated by a single colon.

By using rsync daemon:

Rsync [OPTION] [USER @] HOST: src dest or

Rsync [OPTION] rsync: // [USER @] HOST [: PORT]/SRC [DEST]

Copy files from the remote rsync server to the local machine. The SRC address path is separated by a double colon.

Rsync [OPTION] SRC [USER @] HOST: DEST or

Rsync [OPTION] SRC rsync: // [USER @] HOST [: PORT]/DEST

Copy files from the local machine to the remote rsync server. The DEST address path is separated by a double colon.

If only the SRC parameter is specified in the rsync command and the DEST parameter is not specified, the source file list is displayed instead of the synchronous copy. Rsync has many function options. common options are as follows:

 

 

Table 5. common rsync parameters

 

Option	Description
-A, -- archive	Archive mode to keep all file attributes, equivalent to-rlptgoD
-V, -- verbose	Detailed information output
-R, -- recursive	Recursive processing of sub-directories
-R, -- relative	Use relative path information
-B, -- backup	Create backup
-Z, -- compress	Compressing backup files during transmission
-- Delete	Used to synchronize directories and delete files that do not exist in SRC from DEST.
-- Progress	Show backup process

 

Instance used:

View server files and lists

# Rsync 9.186.110.53 ::

View the list of available modules and comments on the server

# Rsync ibmuser@9.186.110.53: www/

View the directory and file list in the www module of the server (using the rsyncd user authentication method)

# Rsync ibmuser@9.186.110.53:/var/www/html/

View the content in the/var/www/html Directory of the server (use the system user of the server for verification, such as ibmuser)

Data synchronization between the client and the server

# Rsync-avz ibmuser@9.186.110.53: www // backup1/

Back up the content of the www module on the server end to the local/backup1 directory, and retain the original permissions, attributes, owner, and symbolic connections during backup, data transmission is accelerated by compression.

# Rsync-avz ibmuser@9.186.110.53:/var/www/html/backup2/

Use ssh to back up the remote/var/www/html directory to the local/backup2/directory.

# Rsync-avz -- delete ibmuser@9.186.110.53: www // backup3/

Back up the remote www module to the local/backup3/directory, synchronize the directory, and delete unnecessary files from the local directory.

When data on the server is faulty, the server needs to be restored through the client data. as long as the client has the write permission on the server, you can change the SRC and DEST parameters of the rsync command to restore the server.

Author "devilkin"