Introduction and comparison of commonly used file transfer methods under Linux

Reference Link: http://mingxinglai.com/cn/2014/03/copy-file-in-linux/

This paper introduces several ways of transferring files between Linux, and tests the transmission speed between several file transfer modes through specific experiments. This article is an experiment report of my homework, I often check this document, so it is convenient for me to check.

# # #0. Experimental environment and experimental data

Experimental environment: Two computers with Ubuntu, two computers located in the same LAN, the transmission speed of about 4.1mb/s.

Experimental data: Using the MySQL log file (IB_LOGFILE0) for testing, log file compression before 1.1G, compressed 159M, the specific application, the compression ratio may not be so high, but does not affect our discussion.

# # #1. Scp

SCP is the abbreviation of secure copy, SCP is a secure remote file copy command based on SSH login under Linux system, mainly used for copying files and directories between Linux servers. SCP transmits data using SSH security protocol, which has the same authentication mechanism as SSH, so that secure remote copy files can be implemented.

The efficiency of three different uses of SCP is described below. Note: Please prepare SSH before using SCP.

1.1 SCP does not use compression

Copy the local files to the remote server:

scp -P port ufile [email protected]:~/ufile

To copy files from a remote server to a local usage:

scp -P port [email protected]:~/ufile ufile

After testing, SCP does not enable the compression function, the transfer of IB_LOGFILE0 files requires 4:12s.

1.2 Post-compression transmission

SCP transfer Ib_logfile0 file takes so much time because it does not compress the transmitted data, it can compress the file before making a remote copy. As shown below:

tar -zcf ufile.tar.gz ufilescp -P port ufile.tar.gz [email protected]:~/ufile.tar.gzssh -p port [email protected] ‘tar -zxf ufile.tar.gz‘

After testing, manual compression and then transmission, the average of 4 test results is 00:39s.

1.3 SCP Enable compression

Compared with the first method, the second method greatly reduces the transmission time, but it is more troublesome to execute three statements. A simpler approach is as follows:

scp -P port -C ufile [email protected]:~/ufile

-COption enables the compression feature of SSH, as man ssh you can see in the explanation section of the-C option, there is this sentence: "The compression algorithm is the same used by gzip". SSH and gzip Use the same compression algorithm, that is, the second method is almost the same as the third method, but the third method is simpler and more convenient, so it is recommended to use the third method to transfer data.

The third method of transmitting IB_LOGFILE0 cost 00:52s, which takes 10 seconds more than the second method, which is mainly used for compression and decompression of data in 10 seconds.

In addition, it is worth noting that the compression ratio of the modified compression algorithm has little impact on the total data transfer time, due to the higher compression ratio and the need for more compression time.

2. sftp

SFTP is the abbreviation for secure file Transfer protocol, security File Transfer Protocol, which provides a secure encryption method for transferring files. SFTP has almost the same syntax and functionality as FTP, but SFTP is part of SSH that uses encryption to transmit authentication information and transmitted data, so using SFTP is very secure. However, because this kind of transmission uses the encryption, the decryption technology, therefore the transmission efficiency is lower than the ordinary FTP.

Upload data to the server:

echo progress; echo "put ufile"; echo quit) | sftp -o Compression=yes -o Port=port [email protected] -b

Download data from the server:

echo progress; echo "get ufile"; echo quit) | sftp -o Compression=yes -o Port=port [email protected] -b

In my experiment, it took 1:05s to transfer ib_logfile0 files using SFTP. Slightly slower than the SCP.

3. Straightforward using SSH

To use local data transfer to a remote server:

gzip -c ufile | ssh -p port [email protected] ‘gunzip >ufile‘

To transfer a remote server to a local usage

ssh -p port [email protected] "gzip -c ufile" | gunzip -c > ufile

The transmission of IB_LOGFILE0 in this way requires 00:55s, relative to a scp -C few seconds, but not much. However, SCP usage is simpler and requires only one -C parameter instead of manually calling the gzip compression program.

4. NC

Netcat (NC) is a Swiss Army knife in a network tool that can read and write data through TCP and UDP on the network. By combining and redirecting with other tools, you can use it in a variety of ways in your script. It's amazing what you can do with the Netcat command.

What Netcat do is create a link between the two computers and return two data streams, and what you can do after that is your imagination. You can create a server, transfer files, chat with friends, stream streaming media, or use it as a standalone client for other protocols. We only discuss the use of NC for data transmission in this case.

On the server side:

sudo nc -l -p port | tar -zxf - #l 参数用于监听sudo nc -l -p port > ufile

On the client:

tar -zcf - ufile | sudo nc host portsudo nc host port < ufile

Using NC to transfer IB_LOGFILE0 files requires 00:49s, SCP -C which is slightly faster, because the SCP needs to encrypt the data, and NC simply transmits the data.

As you can see, NC does not require any configuration operations and is very simple to use, but the NC requires root permissions.

5. rsync

rsync (remote sync) is a data mirroring Backup tool under Unix-like systems, which can be seen from the name of the software, which is mainly used for synchronous backup of data.

Rsync has two uses, one for transmitting data over an SSH channel and the other for transmitting data by establishing a connection to the server's rsync daemon (daemon) process. The following tests are performed on both of these scenarios.

5.1 Rsync with SSH

Rsync uses SSH channel to transfer data, the configuration is relatively simple, as long as the configuration of SSH, directly available, no need for additional operations, it is this reason, many people prefer to use this way to use rsync.

Here's how to use it:

rsync -zav --rsh=‘ssh -p port‘ ufile [email protected]:path

Rsync uses SSH channel transfer ib_logfile0 to spend 00:55s, just as it does with SSH for data transfer.

5.1 Rsync with Daemon

Unlike SCP and Sftp,rsync, which is a separate set of software, it is possible to transfer data through the daemon process in addition to the SSH channel.

This is just a simple test of the performance of rsync data transmission, and the configuration of rsync can be used to refer to other data.

Rsync is used in the following ways:

rsync -avz ufile [email protected]::module_name

In the experiment, the rsync transmission Ib_logfile0 spent a total of 00:51s, and scp -C almost. In addition, it is worth noting that Rsync uses the zlib compression algorithm to compress data. SSH and gzip use the LZ77 algorithm.

The parameters of rsync are particularly numerous, and the detailed use method can be consulted here.

6. FTP

The FTP command uses the file Transfer Protocol (Files Transfer Protocol, FTP) to transfer files between a local host and a remote host or between two remote hosts.

The use of FTP is no longer introduced, in our experiment, the use of FTP transmission IB_LOGFILE0 Total cost of 4:25s, this is because FTP does not have data compression function. From the previous experimental environment, it can be seen that data compression can significantly reduce the amount of data transferred, the data can not be compressed, and the best transmission tools will be limited to the network transmission rate.

7. Conclusion

Purely from the data transmission, I personally prefer the SCP, but also strongly recommend SCP, however, it is important to remember that use the SCP must use -C the option, that is, the compression feature is enabled. As you can see from the previous experiment, there is a lot of difference between enabling compression and not enabling compression transfer.

In addition, the above several data transmission methods also have their own application scenarios, which, NC Use the simplest, configuration is the most convenient, but requires root permissions, in the case of root permission is not configured SSH, it is recommended to use NC.

The rsync configuration is relatively complex, and its main function is to perform incremental backups rather than data transfers, and without hesitation choose rsync in the case of incremental backups.

FTP configuration is also troublesome, transmission speed is also very general, and, FTP does not have the compression function, need manual compression, but, FTP is the most commonly used data transmission method, most ordinary users have used FTP, for users, learning cost is lower, when the need to transfer files to multiple users, It is recommended to use FTP.

SFTP is built on SSH FTP, transmission speed is very fast, and SCP, but the use of sftp can be interactive, in some cases may be more useful, and, if you have configured the SSH,SFTP configuration cost is zero, that is, without configuration, directly available.

In general, each transmission method is different, we should according to their actual needs, choose the appropriate file transfer method.

sftp (Secure File Transfer Protocol): Security File Transfer Protocol. You can provide a secure encryption method for transferring files. SFTP has almost the same syntax and functionality as FTP. SFTP is a part of SSH and is a secure way to transfer files to the server. In the SSH package, a secure file transfer subsystem called SFTP (secure files Transfer Protocol) has been included, andSFTP itself does not have a separate daemon. It must use the sshd daemon (the port number by default is 22) to complete the corresponding connection operation, so in a sense, sftp is not like a server program, but more like a client program.

OpenSSH: is a free open source implementation of the SSH (Secure SHell) protocol. The SSH protocol family can be used for remote control, or to transfer files between computers. Traditional ways of doing this, such as Telnet (terminal emulation protocol), RCP ftp, rlogin, rsh, are extremely insecure, and passwords are sent in clear text. OpenSSH provides server-side daemon and client tools to encrypt data in remote control and file transfer processes, replacing the original similar services. OpenSSH is the realization of using SSH to encrypt communication through the computer network. It is an open source solution that supersedes the commercial version provided by SSH Communications security. Currently OpenSSH is OpenBSD's sub-plan. OpenSSH is often mistaken for a connection to OpenSSL, but in fact the two programs have different purposes, and different development teams have similar names just because they have the same software development goal-to provide open source encrypted communication software.

ssh (Secure Shell):established by the Network Working group of the IETF, SSH is a security protocol based on the application layer and the transport layer. SSH is currently a more reliable protocol that provides security for Telnet sessions and other network services. The use of SSH protocol can effectively prevent the information leakage in the remote management process.

• SSH is made up of the client and server software: The server is a daemon (daemon) that runs in the background and responds to connection requests from the client. The server is generally the sshd process, which provides the processing of the remote connection, generally including public key authentication, key exchange, symmetric key encryption and unsecured connection; The client contains SSH programs as well as SCP (remote copy), slogin (remote login), Other applications such as SFTP (secure file transfer).
• From the client side, SSH provides two levels of security verification: the first level (password-based security authentication), and the second level (key-based security authentication).
• SSH has three main components: Transport Layer Protocol [Ssh-trans]; user authentication protocol [Ssh-userauth]; Connection Agreement [Ssh-connect].

Introduction and comparison of commonly used file transfer methods under Linux