Several optimization application solutions of rsync

Article Title: Several optimization application solutions of rsync. Linux is a technology channel of the IT lab in China. Includes basic categories such as desktop applications, Linux system management, kernel research, embedded systems, and open source.

Rsync is a good tool for file synchronization. The traditional rsync is to make the files in the two directories consistent, but as the number of files increases, rsync will cause slow synchronization, the system load is high until the system crashes.

There are many solutions to solve the problem of rsync slowdown caused by increasing files.

1. Save fewer files in the source directory

This is a traditional optimization method, because although rsync synchronizes all files, it is the same as synchronizing recently updated files. Therefore, it deletes the directories on the source server, the number of files that have only been updated recently has become small and stable. Over time, the number of files will not increase rapidly. However, rsync cannot use the delete mode. to delete a file, you can empty it. If you have more strict requirements, you can use another program to delete it.

2. Use/dev/shm memory Partition

If the source directory keeps fewer files and the files do not exist on the hard disk and are put into the memory, the system IO problems can be avoided, however, this memory partition will lose all data after the system reboot. Although reboot is not often required, the risks also need to be calculated clearly.

3. Use the push Method

Because the performance problem occurs on the rsync client, if the source server is captured by the production server, the performance problem will occur on the production server, which is of course inappropriate. If rsync daemon is used on the production server, the source server executes the rsync command to push the file to the production server, and the performance problem is transferred to the source server, this ensures the stability of production servers to a certain extent.

4. Use only one instance for Synchronous comparison

If the files on the source server are to be synchronized to many production servers, rsync concurrency may occur. It can be analyzed that the files on these production servers are consistent at the same time. Therefore, it is a waste to compare each server with the source server. At this time, the source server and the production server can be synchronized once, and the-v parameter can be used to print out the log. Other production servers can avoid the comparison process several times by synchronizing the log file.

5. Use inotify

Inotify is not rsync, and inotify is a daemon process that can monitor file changes in the file directory and use rsync for file transmission based on the output, you can delete the file comparison link. Inotify is not complex to use. It monitors file changes in real time without consuming a lot of performance.

6. dual-channel Synchronization

All of the above are optimized for rsync performance, but the optimization will also cause problems. In solutions 3, 4, and 5, if a production server has a server reboot due to load or other problems, synchronization fails during the reboot process, if this part of failed files are not processed, they will never be synchronized to the production server. In this case, multiple rsync methods can be used for processing. For example, inotify is used for real-time synchronization, and then complete rsync synchronization is performed hourly. This ensures a high synchronization speed and limits the risk of losing files within one hour.