Linux operation and architecture-real-time synchronization scheme

1, Inotify+rsync real-time synchronization

1. Introduction

Inotify-tools is a powerful, granular, asynchronous file system event monitoring mechanism that can be used to monitor file system events. Inotify-tools is written in C, except that it requires the kernel to support inotify, and does not depend on others. Inotify-tools provides two tools, one is inotifywait, it is used to monitor file or directory changes, and the second is Inotifywatch, it is used to count the number of file system access.

2. Principle of realization

3, according to the principle of deployment

① See if the system supports inotify

1 0  A :  max_queued_events10 :  Max_user_instances10: max_user_watches

② Software Installation

Wget-o/etc/yum.repos.d/epel.repo http://Mirrors.aliyun.com/repo/epel-6.repo    # Install Epel source to install Yum inotifyyum install-y inotify-tools

③ Test Monitoring Events Create, delete, Close_write, moved_to, Moved_from, Defining inotify monitoring command Parameters

' %w%f ' /data#/data for the monitored directory

④ Scripting for real-time synchronization with rsync and INotify services

Cat >>inotify.sh <<eof#!/bin/sh/usr/bin/inotifywait-mrq--format "%w%f"-e moved_to, delete,create,close_write/data|  while Read Line  Do     -az--delete/data/  [Email protected] 172.16. 1.41:: Nfsbackup--password-file=/etc/rsync.passworddoneeof

⑤inotify optimization

[Email protected] ~]# ll/proc/sys/fs/inotify/-rw-r--r--1Root root0The A  -: -Max_queued_events-->set the number of events that a monitoring service instance can monitor-rw-r--r--1Root root0The -  -: -Max_user_instances-->set the number of service processes that users can open-rw-r--r--1Root root0The A  -: -Max_user_watches-->maximum number of files that can be monitored optimization parameter adjustment: echo "50000000">/proc/sys/fs/inotify/Max_user_watchesecho "50000000">/proc/sys/fs/inotify/max_queued_events

⑥inotifywait Command Parameters

Parameter name	Parameter description
-m,--monitor * * * *	Always keep event listening state
-r,--recursive * * * *	Recursive monitoring of directory data information changes
-q,-quiet * * * *	Less output information (only event information is printed)
--exclude	Case insensitive when excluding files or directories
--TIMEFMT * * * *	Specify the time output format
--format * * * *	Print uses the specified output-like format string to actually monitor the output's content
-E * * *	Specifies the listener's event, if omitted, to indicate that all events are monitored

⑦inotifywait Events Monitoring Event

Event name	Event description
Access	File or directory contents are read
Modify	File or directory contents are written
attrib	File or directory property changes
Close_write * * * *	File or directory is closed and closed after write mode is opened
Chose_nowrite	File or directory is closed after read-only mode is turned off
Close	File or directory close, regardless of read or write mode
Open	File or directory is opened
Moved_to	Files or directories are moved to the monitored directory
Moved_from	The file or directory is moved from the monitored directory
Move * * * *	Files or directories trigger events regardless of whether they move or move out of the monitoring directory
Create * * * *	File or directory created in the monitored directory
Delete	The file or directory is deleted in the monitored directory

2, Sersync real-time synchronization

1. Introduction

Data synchronization will inevitably think rsync that the general simple server data transfer will be used, ftp/sftp but such a method is not efficient, does not support differential incremental synchronization also does not support real-time transmission. For real-time data synchronization needs most people will choose rsync+inotify-tools the solution, but there are some shortcomings, is the Chinese sersync based on the development of the tools, not only the advantages of preserving the benefits of real-time monitoring, file filtering, simplified configuration and other functions to help users improve operational efficiency, Save time and network resources. Https://github.com/wsgzao/sersync Official Information

2. Sersync Advantages

①sersync filters the temporary files and duplicate file operations generated by the Linux system file system so that when combined with rsync, it saves both runtime and network resources, so it is faster.

② compared to the Inotify+rsync,sersync configuration is simple, the bin directory already has basically statically compiled 2 binaries, with the bin directory under the XML configuration file directly to use.

③ in addition Sersync compared to other scripts open source software, using multi-threaded synchronization, especially when synchronizing large files, can ensure that multiple servers in real-time synchronization state.

The ④sersync has an error handling mechanism that synchronizes the failed files through the failure queue and, if it still fails, synchronizes the failed files by the set length.

⑤ comes with the crontab function, just open in the XML configuration file, can be on demand, the whole synchronization once a period of time, no additional configuration crontab function.

⑥sersync has socket and HTTP plug-in extension to meet the needs of two development

3. Realization of schematic diagram

4. Sersync Service Deployment

① software Download Installation

HTTPS://github.com/wsgzao/sersynccd/server/tools//usr/local//usr/local/ sersync/mv Sersync/*  ./rm sersync/-rfchmod +X/USR/LOCAL/SERSYNC/BIN/SERSYNCCP conf/ Confxml.xml  Conf/confxml.xml.bak

② Modifying a configuration file

③ Start Sersync Service

/usr/local/sersync/bin/sersync-dro/usr/local/sersync/conf/confxml.xml

5, high concurrency scene without distributed storage real-time synchronization solution

①inotify (sersync) +rsync system file level

②DRBD File System level

③ synchronization features for third-party software MySQL sync, oracle,mongodb

④ business, through the program double write

⑤ Deprecated NFS Scenario

Linux operation and architecture-real-time synchronization scheme