Crontab +RSYCNC Single-Process low-latency file synchronization

1. Business Requirements: Files uploaded by the website need to be synchronized across the country's servers.
2. Implementation methods：
1). Inotify+rsync. INotify Monitor file changes. Then call rsync to synchronize the changes to the directory. Advantages: Synchronization of file changes, reduce the number of file synchronization, because each synchronization needs to compare the server file list, so this method can reduce the bandwidth consumption. Cons: In a scenario where both servers are frequently uploaded. Uploading multiple files in the same directory will result in more repetitive calls and greatly reduce the efficiency. Of course, you can create a synchronization directory queue. Filter duplicate sync directories to address this disadvantage
2). Crontab+rsync. Timed synchronization. Advantages: Simple implementation, a few lines of code can be achieved, suitable for more file changes in the scene. Cons: Regardless of whether the file has changed or not has been synchronized. Will waste some network bandwidth. Need to avoid concurrency. Synchronization of files is repeated concurrently.

3. Business implementation
1) rsync.sh synchronization script. The script implements checking for a synchronization process every 10 seconds. If not, start the synchronization process, or wait for the synchronization process to be in progress.

#!/bin/bash
#rsync All
#每10秒检查一次, if there is no synchronization process, start the synchronization
Idx=1
While [$idx-le 6];d o
echo "Idx= $idx"
Date
#判断是否有正在同步的进程.
Pid=ps -ef |grep rsync|grep rsyncd.pw|grep imgrsync|awk ‘{print $2}‘

if [ "$pid" ];then    echo "rsync is running,pid:$pid"    sleep 10s     idx=$((idx+1))else    rsync -avzrt --delete  --exclude-from=/opt/rsync/exclude-list.txt --password-file=/etc/rsyncd.pw /path-to-rsync/ rsync://[email protected]{host}:{port}/images    exit 1fi

Done

2）crontab  crontab 保证每1分钟启动一次rsync.sh    `*/1 * * * * /opt/rsync/rsync.sh>/var/log/rsync/rsync-all.log`

Crontab +RSYCNC Single-Process low-latency file synchronization