The art of rsync synchronization

Rsync synchronization art if you are an O & M engineer, you are likely to face dozens, hundreds, or even thousands of servers, in addition to batch operations, environment synchronization and data synchronization are also essential skills. When it comes to synchronization, rsync is a powerful tool that I have to mention. today I will talk about it from this job... if you are an O & M engineer, you may face dozens, hundreds, or even thousands of servers. in addition to batch operations, environment synchronization and data synchronization are also essential skills. When it comes to "synchronization", rsync is a powerful tool that I have to mention. today I will talk about the synchronization art I have seen from this tool. [Without any options] we often use rsync: $ rsync main. c machineB:/home/userB1 data synchronization will be triggered as long as the object content at the target end is different from that at the source end. rsync will ensure that the files on both sides are the same. 2. However, rsync does not synchronize the file's "modify time". for files with data synchronization, the "modify time" of the target file will always be changed to the latest time. 3. rsync does not focus too much on the rwx permission of the target file. if the target does not have this file, the permission will be consistent with that of the source. if the target has this file, the permission will not change with the source. 4. as long as rsync has the read permission on the source file and write permission on the target path, rsync can ensure that the target file is synchronized to the source. 5. rsync can only create files by logging on to the target account. it cannot maintain the consistency between the master and Group of the target file and the source file. (Unless you use the root permission, you are eligible to require consistent owner and group.) [-t option] we often use the-t option: $ rsync-t main. after c machineB:/home/userB1 uses the-t option, rsync always thinks about one thing, that is, synchronizing the "modify time" of the source file to the target machine. 2 rsync with The-t option will become smarter. it will first compare the timestamp and file size of the files on both sides before synchronization. if they are consistent, it will be considered that the files on both sides are the same, this file will no longer be updated. 3. due to rsync's cleverness, it will also lead to smart mistakes. If the timestamp, size, and content of the target file are exactly the same as those of the source file, rsync cannot find them. This is the legendary "pitfall "! 4. when rsync is smart, the solution is to use the-I option. [-I option] we often use the-I option: $ rsync-I main. the c machineB:/home/userB1-I option will make rsync very honest, and it will initiate data synchronization from one file to another. The 2-I option ensures data consistency. The cost is that the speed slows down because the "quick check" policy is abandoned. (The quick check policy is to first check the file timestamp and file size, and first exclude a batch of files that are considered to be the same) 3 whatever the situation, the modify time of the target object is always updated to the current time. [-V option] This option is easy to understand. rsync outputs more information. for example, $ rsync-vI main. c machineB:/home/userB main. c sent 81 bytes converted Ed 42 bytes 246.00 bytes/sectotal size is 11 speedup is 0.09 the more v you add, the more log information you get. $ Rsync-vvvvt abc. c machineB: /home/userB cmd = machine = machineB user = path =/home/userBcmd [0] = ssh cmd [1] = machineB cmd [2] = rsync cmd [3] = -- server cmd [4] =-vvvvte. cmd [5] =. cmd [6] =/home/userB opening connection using: ssh machineB rsync -- server-vvvvte .. /home/userB note: iconv_open ("ANSI_X3.4-1968", "ANSI_X3.4-1968") succeeded. (Client) Protocol versions: remote = 28, negotiated = 28 (Server) Protocol vers Ions: remote = 30, negotiated = 28 [sender] make_file (abc. c, *, 2) [sender] flist start = 0, used = 1, low = 0, high = 0 [sender] I = 0 abc. c mode = 0100664 len = 11 flags = 0send_file_list donefile list sentsend_files startingserver_recv (2) starting pid = 31885recv_file_name (abc. c) received 1 names [Explorer] I = 0 abc. c mode = 0100664 len = 11recv_file_list doneget_local_name count = 1/home/userBrecv_files (1) startinggenerator sta Rting pid = 31885 count = 1 delta transmission enabledrecv_generator (abc. c, 0) abc. c is uptodategenerate_files phase = 1send_files phase = 1recv_files phase = 1generate_files phase = 2 send files finishedtotal: matches = 0 hash_hits = 0 false_alarms = 0 data = 0generate_files finishedrecv_files waiting on 14318 sent 36 bytes encoded Ed 16 bytes 104.00 bytes/sectotal size is 11 speedup is 0.21 _ exit_cleanu P (code = 0, file = main. c, line = 1031): entered_exit_cleanup (code = 0, file = main. c, line = 1031): about to call exit (0) [-z option] this is a compression option. if this option is used, rsync first compresses the data sent to the peer end and then transfers the data. If the network environment is poor, we recommend that you use it. Generally, the-z compression algorithm is the same as that of gzip. [-R option] when we use rsync for the first time, we often encounter this problem? Atlas? /Div> $ rsync superman machineB:/home/userBskipping directory superman if you don't tell rsync that you need it to help you synchronize folders, it won't take the initiative, this is also the lazy aspect of rsync. Therefore, if you really want to synchronize folders, add the-r option, that is, recursive (recursive, cyclic), like this: $ rsync-r superman machineB: /home/userB as we mentioned above, if the timestamp and file size are exactly the same, only the file content is different, and you do not use the-I option, then, rsync does not synchronize data. In the Linux World, folders are Files. if these files (folders) have different contents, the timestamp and file size are the same, will rsync find it?" You can do this experiment by yourself. The conclusion is as follows: rsync will be clear for folders, as long as you add the-r option, it will do its utmost to go to the folder to check, instead of simply "quick check" for the folder itself. [-L option] if we want to synchronize a soft link file, what will rsync prompt you? $ Lltotal 128-rw-rw-r -- 1 userA 11 Dec 26 07:00 abc. clrwxrwxrwx 1 userA 5 Dec 26 :35 softlink-> abc. c $ rsync softlink machineB:/home/userBskipping non-regular file "softlink" Well, you guessed it, and rsync refused us relentlessly. Once it finds that a file is a soft link, it will ignore it unless we add the-l option. $ Rsync-l softlink machineB:/home/userB after the-l option is used, rsync will completely retain the soft link file type, and copy the soft link file to the target in the original version, instead of "follow link" to the object file. If I want rsync to adopt the follow link method, use the-L option. You can try the effect on your own. [-P option] the full name of this option is "perserve permissions". as the name suggests, it means to keep the permission. If you do not use this option, rsync handles permission issues like this: 1. if the target end does not have this file, after synchronization, the permissions of the target file will be consistent with those of the source. 2. if the target file already exists, only the file content will be synchronized, and the permissions will remain unchanged. If you use the-p option, rsync keeps the target end consistent with the source end. [-G option and-o option] these two options are a pair, which is used to maintain the group and owner of the file. The role should be clear and clear. However, you must note that only administrators can change the owner and group permissions. [-D option]-D option. The original text is "preserve devices (root only)". literally, it means to keep the original information of the device file. As bloggers have never actually experienced the benefits of it, they have no right to speak. [-A option] 1-a option is a more aggressive option in rsync, because you use the-a option, which is equivalent to using the-rlptgoD option. The only option is-. (After reading the previous article, you should be able to easily understand the functions of the seven options) 2-a option should be named archive option, while Chinese is called archive option. Using the-a option indicates that you want to use a recursive method for synchronization and maintain consistency in all aspects as much as possible. 3. However, The-a option also has the Achilles heel, that is,-a cannot synchronize "hard links. If you have such requirements, add the-H option. [-- Delete option, -- delete-excluded option, and -- delete-after option] all three options are related to "delete": 1-delete: if the source end does not have this file, then, you can delete it without having to own it. (If you use this option, you must use the-r option together.) 2-delete-excluded: specify objects to be deleted on the target. 3-delete-after: by default, rsync clears the target file before data synchronization. if this option is used, rsync synchronizes data first, and then delete the files to be cleared. If you see so many delete operations, do you have a trembling liver? Indeed, in the official instructions of rsync, This option can be dangerous if used incorrectly! It is a very good idea to run first using the dry run option (-n) to see what files wocould be deleted to make sure important files aren't listed. in this sentence, we learned a trick, that is, the-n option, which is an option to scare people and will warn you with the list of affected files, but it won't be deleted, which gives us the opportunity to confirm and maneuver. Let's take a look at the actual usage: $ rsync-n -- delete-r. machineB:/home/userB/deleting superman/xxxdeleting main. cdeleting acclink [-- exclude option and -- exclude-from option] if you do not want to synchronize something to the target end, you can use the-exclude option to hide it, rsync attaches great importance to privacy. you can use the-exclude option multiple times to set a lot of "privacy ". If you want to hide too much privacy, setting it in the command line option will be troublesome, and rsync is still considerate. It provides the-exclude-from option, this allows you to list privacy in one file, and then let rsync directly read the file. [-- Partial option] this is the legendary resumable Upload function. By default, rsync deletes the files that are interrupted by transmission and re-transmits the files. However, in some special cases, we do not want to re-transmit data, but resume data transmission. We often see people using the-P option, which is designed to be lazy. In the past, people always had to manually write-partial-progress. they thought it was too difficult to replace it with a new option, so-P came into being. Some readers may ask-partial: I know the role, but what is progress used? Why is it so attractive for many people to use it? (Actually ...) [-- Progress option] when This option is used, rsync will display the transmission progress information. what is the purpose? rsync provides an interesting explanation: This gives a bored user something to watch. well, after writing so much, everyone is reading it very boring. it is a good choice to actually use-progress to relieve boredom: ^_^ PS: the PATTERN syntax for rsync-exclude will be explained later. Thank you!