Linux file synchronization

How to Copy the Files across a network/internet in Unix/linux (Redhat, Debian, FreeBSD, etc)-SCP tar rsync

One of the many advantages of Linux/unix is and how many ways your can do one thing. This tuturial are going to show you some of the many ways your can ttransfer files over a network connection.

In this article/tutorial we'll cover rsync, SCP, and tar. Please note this there are many other ways this are just some of the more common ones. The methods covered assume that SSH are used in all sessions. These methods are all the more secure and reliable than using RCP or FTP. This tutorial is a great alternative for those looking for a FTP alterative to transfering files over a network.

Scp

SCP or secure copy is probably the easiest of the methods, the it is designed as a replacement to RCP, which was a quick Copy of CP with Network funcationability.

SCP syntax

SCP [-CR]/some/file [more ...] host.name:/destination/file

-or-

SCP [-CR] [[User@]host1:]file1] [[User@]host2:]file2

Before SCP does any copying it-connects via SSH. Unless proper keys are in place, then you would be asked for usernames. You can test if it is working by using ssh-v hostname

The-r switch is used when you want to recursively go through directories. Please note your must specify the source file as a directory to work.

SCP encrypts data over your network connection, but by using the-c switch you can compress the data before it goes over T He network. This can significantly decrease the time it takes to copy large files.

Tip:by default SCP uses 3DES encryption algorithm, all encryption algorithms are slow, but some are faster than others. Using-c Blowfish can speed things up.

What SCP shouldn ' t be used for:
1. When you are copying the than a few files, as SCP spawns a new process for each file and can is quite slow and RESOURC e intensive when copying a large number of files.
2. When using the-r switch, SCP does not know about symbolic links and'll blindly follow, them if it even has M Ade a copy of the file. The can leads to SCP copying a infinite amount of data and can easily fill up your disk, so is hard.

Rsync

Rsync has very similar syntax to SCP:

RSYNC-E ssh [-avz]/some/file [more ...] host.name:/destination/file

-or-

Rsync-avce SSH Source.server:/path/to/source/destination/dir

Rsync's speciality lies in its ability to analyse files and only copy of the changes made to files rather all files. This can leads to enormous improvements when copying a directory of second time.

Switches:

-a Archive mode, most likely you should always keep this. Preserves file permissions and does not follow symlinks.

-v Verbose, lists files being copied

-Z Enable compression, this'll compress each file as it gets sent over the pipe. This can greatly decrease time depending on what sort files for you are copying.

-e ssh Uses ssh as the transport, this should always to be specified.

-C exclude CVS file eg: *.o *.a *.old *. bak *.rej Tags core

Disadvantages of using rsync:
1. Picky syntax, use of the trailing slashes can be confusing.
2. Have to remember so you are using SSH.
3. Rsync is not installed to all computers.

Tar

The tar is usually used for achiving applications, but what we are going to does in the this case are tar it then pipe it over a ssh Connection. Tar handles large file trees quite and preserves all file permissions, etc, including those UNIX systems which use AC Ls, and works quite symlinks.

The syntax is slightly different as we are piping it to ssh:

TAR-CF-/some/file | SSH host.name TAR-XF--c/destination

-or with compression-

TAR-CZF-/some/file | SSH host.name Tar-xzf--c/destination

Switch-c for Tar creates a archive and-f which tells Tar to send the new archive to stdout.

The second tar command uses THE-C switch which changes directory on the target host. It takes the input from stdin. The-x switch extracts the archive.

The second way of doing the transfer over a network are with THE-Z option, which compresses the stream, decreasing time it Would take to transfer over the network.

Some people may ask why be used, this is great for large file trees, as it's just streaming the data from one host to Another and not has to do intense operations with file trees.

If using the-v (verbose) switch, be sure only to include it in the second tar command, otherwise you'll double OUTP Ut.

Using tar and piping can also is a great way to transfer files locally to be sure this file permissions are kept correctly :

Tar CF-/some/file | (Cd/some/file, Tar XF-)

This could seem like a long command, but it are great for making sure all file permissions are kept in tact. What It is doing are streaming the files in a sub-shell and then untarring them in the target directory. Please note that the-z command should is used for local files and no perfomance increase'll be visible as overhead Processing (CPU) would be evident, and would slow down the copy.

Why tar shouldn ' t be used:
1. The syntax can be hard to remember
2. It ' s not as quick as to type SCP for a small number of files
3. Rsync would beat it hands down for a tree of the files of this already exist in the destination.

There are several ways of copying over a network, such as FTP, NAS, and NFS but this all requre specialised e installed on either the receiving or sending end, and hence are not as useful as the above.