Real-time synchronization of data files (rsync + sersync2)

Due to the recent project needs, synchronization of cloud server data to the * * side to make big data analysis.

Ideas:

Initially, as long as the data is synchronized, prepare an Open data acquisition interface. But the real-time is poor, there will be delays.

Therefore, the search for various solutions, finally determined to use rsync for file synchronization, otter (Ali's Open source tools) for database synchronization

The results of the successful use of synchronization is significant!

This article only describes the Rsync file synchronization, Otter will share in other articles

First, Introduction

A bunch of online introduction, but all written very official, the following introduction is my own use after the feeling.

Sersync is used to monitor the directory after invoking the rsync Synchronization command for file transfer! can monitor the target directory files to increase, delete, change

Rsync image Backup tool, can quickly achieve file synchronization, file synchronization of the core! Supports local replication, or synchronizes with other SSH and rsync hosts.

Environment Description:

System Environment: CENTOS7

Database version: MySQL 5.7

Purpose: Synchronize source server files and data to the backup server in real time

Installation Policy

1、目标服务器安装 rsync 服务，设置传输项目名、用户名、密码、端口2、源服务器安装 rsync ，测试传输命令3、源服务器安装 sersync ，配置监控目录

The source server does not need to be configured, just write a password file for the rsync command to execute using the target server configuration as follows:

# 端口 port=873   # 安全防护                  use chroot = yes# 最大并发数max connections = 36000 # rsync 的守护进程将其 PID 写入指定的文件。pid file=/var/run/rsyncd.pid # 指定支持 max connections 参数的锁文件。lock file=/var/run/rsync.lock # 指定 rsync 守护进程的日志文件，而不将日志发送给 syslog。log file=/var/log/rsyncd.log # 任务名称[taskName] # 存储路径 (需要手动创建,无目录会报错) path=/data/Vehicle/image/ # 指定是否允许客户上传文件。若为 true 则不允许上传；若为 false 并且服务器目录也具有读写权限则允许上传。read only=false # 指定当客户请求列出可以使用的模块列表时，该模块是否应该被列出。如果设置该选项为 false，可以创建隐藏的模块。list=yes # 用户名auth users= 用户名 # 密码文件secrets file=/etc/rsync.passwd # 用一个主机列表指定哪些主机客户允许连接该模块。不匹配主机列表的主机将被拒绝hosts allow=*    

About password files

such as/ETC/RSYNC.PASSWD

Source server:

密码

Target server:

用户名：密码

The password file must have execute permissions of 600, or the service will not start

Modify Permissions Command:

chmod 600 /etc/rsync.passwd  

Rsync Transfer command

Once you have installed the above services and turned on the Rsync service on the target server, you can test the synchronization of the files.

The rsync Synchronization command is as follows:

rsync -avzP --delete 传输目录 用户名@目标服务器ip::任务名称 --password-file = 密码文件地址

-V Show Instruction execution process

-Z Compress the backed-up files as they are transmitted

-P preserves the properties of the source file or directory.

--delete Delete the transfer end no longer exists, and the destination exists on the file

Once the test is successful, the directory can be transferred to the target server, and the next step is to install the SERSYNC2 service to monitor the directory changes that you want to transfer, so that you can achieve real-time synchronization

About the download, no official website, I was looking for on the CSDN, everybody refueling Bar

The configuration file is as follows:

<?xml version= "1.0" encoding= "iso-8859-1"?> 

Sersync Start command

进入 sersync安装目录./sersync -d -r -o confxml.xml

This article does not detail the download installation process, mainly for the introduction of configuration Parameters! Easy to remember later.

Excuse me!

Real-time synchronization of data files (rsync + sersync2)