The Art of rsync syncing

Transferred from: http://roclinux.cn/?p=2643

If you are an OPS engineer, you are likely to face dozens of, hundreds of or even thousands of servers, in addition to batch operations, environment synchronization, data synchronization is also an essential skill.

When it comes to "syncing," the one thing you have to mention is rsync, and today it's the art of syncing that I see from this tool.

[without any options]

We often use rsync this way:

$ rsync main.c Machineb:/home/userb

1 data synchronization is triggered as long as the file content of the destination is different from the source, and rsync ensures that the file content is the same on both sides.
2 but Rsync does not synchronize the "modify time" of the file, and any files with data synchronization, the "Modify times" of the destination files will always be modified to the latest moment.
3 rsync does not focus too much on the rwx permissions of the destination file, and if the destination does not have this file, then the permissions remain consistent with the source, and if the destination has this file, the permissions do not change with the source.
4 as long as rsync has read access to the source file and has write access to the target path, rsync ensures that the destination file is synchronized to the source.
5 rsync can only create files with the login destination account, it does not have the ability to keep the destination files and the host group and the source side consistent. (unless you are using root privileges, you are eligible to request a consistent, group-consistent)

[-t option]

We often use the-t option like this:

$ rsync-t main.c Machineb:/home/userb

1 after using the-t option, rsync always thinks of one thing, which is to synchronize the "Modify time" of the source file to the target machine.

2 rsync with the-t option will become smarter, it will compare the time stamp and file size of both sides of the file before synchronizing, and if it is consistent, it will be considered as a file on both sides, no more update action is taken on this file.
3 because the clever of rsync, also will be clever mistake. If the timestamp, size, and source side of the destination file are exactly the same, but the content happens to be inconsistent, rsync is not found. This is the legendary "pit"!
4 for the rsync smart situation, the solution is to use the-I option.

[-I option]

We often use the-I option in this way:

$ rsync-i main.c Machineb:/home/userb

The 1-i option will make rsync very good and honest, it will be a file to initiate data synchronization.
The 2-I option ensures data consistency, at the expense of slower speed because we abandon the "quick check" policy. (Quick check policy is to check the timestamp and file size of the file first, and then exclude a batch of files that are considered identical)
3 Regardless of the situation, the modify time of the destination file is always updated to the current moment.

"-V Option"

This option, easy to understand, is to let rsync output more information, we can give an example:

$ rsync-vi main.c machineb:/home/userb                                                   main.csent Bayi bytes  received bytes  246.00 bytes/sectotal size is 11  Speedup is 0.09

The more v you add, the more log information you can get.

$ RSYNC-VVVVT abc.c machineb:/home/userb cmd= machine=machineb user= path=/home/userbcmd[0]=ssh Cmd[1]=machineB cmd[2]= Rsync cmd[3]=--server Cmd[4]=-vvvvte. cmd[5]=. Cmd[6]=/home/userb Opening connection Using:ssh Machineb rsync--server-vvvvte. . /home/userb Note:iconv_open ("ansi_x3.4-1968", "ansi_x3.4-1968") succeeded. (Client) Protocol versions:remote=28, negotiated=28 (Server) Protocol versions:remote=30, Negotiated=28[sender] Make_file ( abc.c,*,2) [sender] Flist start=0, used=1, low=0, High=0[sender] i=0 abc.c mode=0100664 len=11 flags=0send_file_list donef Ile list Sentsend_files Startingserver_recv (2) starting Pid=31885recv_file_name (ABC.C) received 1 Names[receiver] I=0 AB C.C mode=0100664 len=11recv_file_list doneget_local_name count=1/home/userbrecv_files (1) Startinggenerator starting pid=31885 Count=1delta transmission Enabledrecv_generator (abc.c,0) abc.c is Uptodategenerate_files phase=1send_files Phase=1recv_files phase=1generate_files phase=2send Files Finishedtotal:maTches=0 hash_hits=0 false_alarms=0 data=0generate_files finishedrecv_files finishedclient_run waiting on 14318sent TEs received bytes 104.00 bytes/sectotal size is one speedup is 0.21_exit_cleanup (code=0, FILE=MAIN.C, line=1031): En Tered_exit_cleanup (code=0, FILE=MAIN.C, line=1031): About-to-call exit (0)

[-Z option]

This is a compression option, and as long as this option is used, rsync will compress and retransmit the data sent to the peer. It is recommended to use in the case of poor network environment.

In general, the-Z compression algorithm will be the same as gzip.

[-R Option]

When we use rsync for the first time, we often encounter such a dilemma:

$ rsync Superman machineb:/home/userbskipping directory Superman

If you don't tell rsync that you need it to sync your folders, it won't take the initiative, which is where rsync is lazy.

So, if you really want to sync folders, then add the-r option, which is recursive (recursive, cyclic), like this:

$  rsync-r Superman Machineb:/home/userb

As we said in the above explanation, if the timestamp is exactly the same as the file size, only the file content is different and you do not use the-i option, then rsync does not synchronize the data.

So, ask a question: "Because in the world of Linux, folders are also files, and if such files (folders) have only the same content, and timestamps and file sizes are the same, will rsync find out?" ”

Experiment you can do it yourself, the conclusion here to tell you:

For folders, Rsync is perspicacious, as long as you add the-r option, it will be dedicated to go into the folder to check, and not only the folder itself to do "quick check."

[-l option]

If we want to sync a soft link file, what do you think rsync will prompt?

$ lltotal 128-rw-rw-r--  1 userA userA  1 abc.clrwxrwxrwx userA  5 Dec userA 11:35 Softlink Gt abc.c$ rsync softlink machineb:/home/userbskipping non-regular file "Softlink"

Well, you guessed it, and Rsync rejected US mercilessly. Once it discovers that a file is a soft link, it ignores it unless we add the-l option.

$ rsync-l Softlink Machineb:/home/userb

With the-l option, Rsync retains the soft-link file type completely, copying the soft-link file to the destination without "follow link" to the pointing entity file.

If I just want to let rsync take follow link way, then use the-l option is possible. You can try the effect on your own.

[-P option]

The full name of this option is "Perserve permissions", as the name implies, is to maintain permissions.

If you don't use this option, Rsync handles permissions issues like this:

1 If the destination does not have this file, then after the synchronization will be the destination file permissions to maintain consistent with the source side;
2 If this file already exists on the destination, only the contents of the file are synchronized, and the permissions remain unchanged.

If you use the-P option, rsync will keep the destination consistent with the source's permissions anyway.

[-G option and-o option]

These two options are a pair that is used to keep the file's group and owner, which should be clearly understood. One thing to note, however, is that changing the owner and the group is often only possible with administrator privileges.

[-D option]

The-D option, the original interpretation is "preserve devices (root only)", in the literal sense, is to keep the original information of the device files. Since the blogger has not actually experienced its benefits, there is not much to say.

[-A option]

The 1-A option is a more overbearing option in rsync because you use the-a option, which is equivalent to using the-rlptgod option. With an enemy seven, the only-a option is also. (after reading the previous article, you should be able to easily understand the effect of these seven options)
The scientific name of the 2-A option should be called archive option, and Chinese is called archive options. Using the-a option indicates that you want to synchronize in a recursive manner and to maintain consistency in all aspects as much as possible.
3 But the-a option also has Achilles heel, that is,-a cannot sync "hard link" situations. If this is a requirement, add the-H option.

[–delete Options, –delete-excluded options, and –delete-after options]

All three options are related to "delete":

1–delete: If the source side does not have this file, then the destination also do not want to have, delete it. (If you use this option, it must be paired with the-R option)
2–delete-excluded: Specifically specify some files to be deleted on the destination side.
3–delete-after: By default, rsync is to clean up the destination file before starting data synchronization, and if you use this option, Rsync will synchronize the data before deleting the files that need to be cleaned.

See so many delete, do you have a little liver fibrillation? Indeed, in the official statement of Rsync, there is a sentence:

This option can be dangerous if used incorrectly!  It's a very good idea to run first using the dry  run  option (-N) to see what files would be deleted to make sure I mportant files aren ' t listed.

In this sentence, we learned a little trick, that is the-n option, it is a frightening option, it will be affected by the list of files to warn you, but will not really delete, which gives us the opportunity to confirm and manoeuvre. Let's take a look at the actual usage:

$ rsync-n--delete-r. Machineb:/home/userb/deleting superman/xxxdeleting main.cdeleting Acclink

[–exclude options and –exclude-from options]

If you do not want to sync something to the destination, you can use the –exclude option to hide it, rsync still attaches great importance to privacy, you can use the –exclude option multiple times to set a lot of "privacy."

If you want to hide too much privacy, setting in command-line options will be cumbersome, rsync is very thoughtful, it provides the –exclude-from option, so you can put the privacy one by one in a file, and then let rsync directly read this file just fine.

[–partial Options]

This is the legendary function of the continuation of the breakpoint. By default, Rsync deletes the files that were interrupted by the transmission and then retransmits them. But in some special cases, we do not want to re-preach, but to continue to preach.

In our use, we often see someone using the-p option, which is actually designed to be lazy. Before people always have to write –partial–progress manually, feel too laborious, rather than with a new option to replace, so-P came into being. Some readers will ask –partial I know the role, but –progress what is the use of it? Why does a lot of people want to use it, and it has so much appeal? (There is really ...) ）

[–progress Options]

With this option, rsync will show the transfer progress information, what's the use of it, Rsync gives a very interesting explanation:

This gives a bored user something to watch.

Well, write so much, we see already very boring, to actually use –progress to relieve the boredom, is a good choice ^_^

PS: Follow-up will explain the –exclude pattern syntax for rsync.

Thank you!

The Art of rsync syncing