Real-time Data Backup Using rsync + inotify

650) This. width = 650; "src =" http://s3.51cto.com/wyfs02/M02/38/17/wKiom1OyhAuDcVADAABNuIyGy5A162.png "Title =" fenfa. PNG "alt =" wkiom1oyhaudcvadaabnuiygy5a162.png "/>

I. rsync

Compared with traditional CP and tar backup methods, rsync has the advantages of high security, fast backup, and support for Incremental backup. Rsync can meet the demand for data backup with low real-time performance, for example, regularly back up the data on the file server to the remote server, and regularly perform data mirroring on the local disk.

As the scale of application systems continues to expand, rsync also puts forward better requirements on data security and reliability. Rsync gradually exposes many shortcomings in high-end business systems. First, when rsync synchronizes data, you need to scan all the files for comparison and perform the differential transmission. If the number of files reaches millions or even tens of millions, scanning all files will be very time-consuming. And the changes are often a small part, which is very inefficient. Secondly, rsync cannot monitor and synchronize data in real time. Although it can trigger Synchronization Through the Linux daemon, there will be a time difference between the two triggering actions, as a result, the data on the server and the client may be inconsistent and the data cannot be completely restored when the application fails. For the above reason, the combination of rsync and inotify has appeared!

2. inotifyinotify is a powerful fine-grained and asynchronous file system event monitoring mechanism. The Linux kernel has been added to inotify since 2.6.13, inotify can be used to monitor the addition, deletion, modification, movement, and other minor events in the file system. With this kernel interface, third-party software can monitor various file changes in the file system, inotify-tools is such a third-party software.

In the previous chapter, we have mentioned that rsync can implement triggered file synchronization, but trigger through the crontab daemon process. The synchronized data is different from the actual data, inotify can monitor various changes in the file system. When there are any changes to the file, rsync synchronization is triggered, which just solves the problem of real-time data synchronization.

3. Install inotify-tools because inotify requires support from the Linux kernel. Before installing inotify-tools, check whether the Linux kernel has reached 2.6.13 or later. If the Linux kernel is earlier than 2.6.13, you need to re-compile the kernel to support inotify. You can also use the following method to determine whether the kernel supports inotify:

[[Email protected] webdata] # ll/proc/sys/fs/inotify

-RW-r -- 1 Root 0 04-13 19:56 max_queued_events

-RW-r -- 1 Root 0 04-13 19:56 max_user_instances

-RW-r -- 1 Root 0 04-13 19:56 max_user_watches

If the preceding three outputs are available, inotify is supported by the system by default, and inotify-tools can be installed.

You can download the corresponding inoy-y-toolsplugin at http://inotify-tools.sourceforge.net/. then, start the compilation and installation:

[[Email protected] ~] # Tar zxvf inotify-tools-3.14.tar.gz [email protected] ~] # Cd inotify-tools-3.14

[[Email protected] inotify-tools-3.14] #./configure

[[Email protected] inotify-tools-3.14] # Make

[[Email protected] inotify-tools-3.14] # make install

[[Email protected] inotify-tools-3.14] # ll/usr/local/bin/inotifywa *

-Rwxr-XR-x 1 Root 37264 04-14 13:42/usr/local/bin/inotifywait

-Rwxr-XR-x 1 Root 35438 04-14 13:42/usr/local/bin/inotifywatch

After inotify-tools is installed, the inotifywait and inotifywatch commands are generated. inotifywait is used to wait for a specific event in a file or file set. It can monitor any file and directory settings, the entire directory tree can be monitored recursively.

Inotifywatch is used to collect statistics on the monitored file system, including the number of times each inotify event occurs.

4. inotify-related parameters inotify defines the following interface parameters that can be used to limit inotify's consumption of kernel memory. Because these parameters are all memory parameters, you can adjust the size of these parameters in real time based on application requirements. The following is a brief introduction.

/Proc/sys/fs/inotify/max_queued_evnets

Indicates the maximum number of events that can be queued in the inotify instance when inotify_init is called. Events that exceed this value are discarded,

In_q_overflow event.

/Proc/sys/fs/inotify/max_user_instances

The maximum number of inotify instatnces that can be created by each real User ID.

/Proc/sys/fs/inotify/max_user_watches

The maximum number of directories that each inotify instatnces can monitor. If the number of monitored files is large, you need to increase the size of this value as appropriate, for example:

Echo 30000000>/proc/sys/fs/inotify/max_user_watches

5. inotifywait-related parameters inotifywait is a monitoring wait event, which can be used with shell scripts. The following describes some common parameters:

-M, that is, -- monitor, indicates that the event listening state is always maintained.

-R, that is, -- Recursive, indicates the recursive query directory.

-Q, that is, -- quiet, indicates that the monitoring event is printed.

-E, that is, -- event. You can use this parameter to specify the event to be monitored. Common events include modify, delete, create, and attrib.

For more information, see man inotifywait.

[Email protected]: ~ $ More/etc/rsyncd. conf

Uid = nobody

Gid = nobody

Use chroot = No

Max connections = 10

Strict modes = Yes

PID file =/var/run/rsyncd. PID

Lock file =/var/run/rsync. Lock

Log File =/var/log/rsyncd. Log

[Web1]

Path =/data/www/

Comment = web1 File

Ignore errors

Read Only = No

Write only = No

List = false

Uid = root

Gid = root

Auth users = web1

Secrets file =/etc/rsync. Password

[Email protected]: ~ $ More/etc/rsyncd. conf

Uid = nobody

Gid = nobody

Use chroot = No

Max connections = 10

Strict modes = Yes

PID file =/var/run/rsyncd. PID

Lock file =/var/run/rsync. Lock

Log File =/var/log/rsyncd. Log

[Web2]

Path =/data/www/

Comment = web2 File

Ignore errors

Read Only = No

Write only = No

List = false

Uid = root

Gid = root

Auth users = web2

Secrets file =/etc/rsync. Password

Web3 web4 web5 web6 .....

[Email protected]: ~ $ More auto_rsync.sh

#! /Bin/sh

Web = (web1 web2 web3 web4 web5 web6)

Src =/data/www/

P_file =/etc/server. Pass

/Usr/local/bin/inotifywait-mrq -- timefmt '% d/% m/% Y % H: % m' -- format' % T % w % F % e'-e modify, delete, create, attrib $ SRC \

| While read files

Do

For J in $ {web [*]}

Do

# Echo 'date "+ % Y-% m-% d % H: % m "'

Echo

Echo $ J

Echo $ files

/Usr/bin/rsync-vzrtopg -- delete -- Progress -- password-file = $ p_file $ SRC [email protected] $ J: $ J

Echo

Echo

Done

Done

Nohup./auto_rsync.sh 2> 1 nohup. Out & start listening

This article is from the "crazy_linux" blog, please be sure to keep this http://mkernel.blog.51cto.com/8015041/1433185