"Linux Tutorial" INotify usage Introduction and combining rsync to achieve real-time file synchronization between hosts

I. Introduction of INotify

INotify is a new subsystem (API) for the Linux kernel 2.6.13 (June 18, 2005), which provides a mechanism for monitoring file system (Inode-based) events to monitor changes in file systems such as file modifications, additions, deletions, etc. The appropriate event can be notified to the application. The mechanism is introduced by the famous Desktop Search engine project Beagle to replace dnotify with similar functionality but with many flaws.

INotify can monitor files as well as monitor directories. When the directory is monitored, it can simultaneously monitor directories and subdirectories and files in the directory. In addition, INotify uses file descriptors as interfaces, so you can monitor file system changes using the usual file I/O operations Select, poll, and Epoll.

Common file system events that inotify can monitor include:

In_access: File is accessed

In_modify: File is modified

In_attrib, file attributes are modified

In_close_write, file opened in writable mode is closed

In_close_nowrite, file opened in non-writable mode is closed

In_open, file is open

In_moved_from, files are removed from the monitored directory

In_moved_to, files are moved into the monitored directory

In_create, create a new file or subdirectory in the monitored directory

In_delete, file or directory is deleted

In_delete_self, self-deletion, that is, an executable file deletes itself when executed

In_move_self, self-moving, that is, an executable file moves itself at execution time

Use the following parameters in the/proc interface to set the amount of memory the inotify is capable of using:

1,/proc/sys/fs/inotify/max_queue_events

When an application calls INotify, it needs to initialize the INotify instance, and when it sets an event queue, the value in this file is used to set the upper limit for this queue length, and events beyond this limit will be discarded;

2,/proc/sys/fs/inotify/max_user_instances

The values in this file are used to set the maximum number of inotify instances that can be created per user ID (the user identified by ID);

3,/proc/sys/fs/inotify/max_user_watches

The values in this file are used to set the maximum number of files or directories that can be monitored by each user ID;

Second, Inotify-tools

INotify is an API that needs to be called through the development of applications, and for most users there is a lot of inconvenience, and Inotify-tools's appearance makes up for this shortcoming. Inotify-tools is a set of components that includes a C library and several command-line tools that can be used to monitor the events of a file system through a command line or script. It was developed by Rohan McGovern, whose project site is http://inotify-tools.sourceforge.net.

Inotify-tools provides two command-line tools:

Inotifywait: The INotify API waits for the corresponding event on the monitored file and returns the monitoring result, by default the normal result is returned to the standard output, and the diagnostic class information is returned to the standard error output. It can exit after monitoring the event specified on the corresponding monitoring object, and can also be monitored continuously.

Inotifywatch: Collects related events and outputs statistics for monitored files or directories through the INotify API.

Introduction to inotifywait command use:

Inotifywait is especially useful for waiting for an event to occur in a script and to perform the appropriate action based on a specific event. If it is used in a script to monitor changes in files in a specified directory, create new, delete, attribute information, and then use the rsync command to synchronize the files corresponding to an event to another host. The common options are as follows:

-M, the default action of--monitor:inotifywait is to exit after a specific event that is monitored to the specified file occurs once, while using this option allows for continuous monitoring;

-R,--recursive: recursively monitors all files in the specified directory, including newly created files or subdirectories, and if the number of files in the directory to be monitored is large, you usually need to modify the/proc/sys/fs/inotify/max_users_watchs kernel parameters. Because its default value is 8192.

-e <event>--event <event>: Specifies the specific events to monitor, by default, all events are monitored; Here <event> includes access, modify, attrib, close_write, Close_nowirte, close, open, moved_to, Moved_from, move, create, delete, Delete_selt, etc.;

--timefmt <fmt>: When%t is used in the--format option, the--TIMEFRT option can be used to specify a custom time format that conforms to the STRFTIME specification, which can be obtained by using the Strftime Manual for the format available in this time format ; the parameters commonly used after--timefrt are '%d/%m/%y%h:%m ';

--format <fmt>: Custom inotifywait output format, such as--format '%T%w%f '; the usual format characters are as follows:

%w: Displays the file name of the monitored files;

%f: If the object of an event is a directory, the name of the monitored directory is displayed, and the default display is an empty string;

%T: Use the custom time format in the--TIMEFMT option;

For example, to monitor the Create,delete,modify,close_write events that occur on the/tmp/test directory and all of its internal files, use the following command:

# inotify-r--timefmt '%d/%m/%y%h:%m '--format '%T%w%f '-e create,delete,modify,close_write/tmp/test

This command exits after monitoring the first event on a file, and if you want to monitor it all the time, you need to add the-m option to the command.

In many scenarios, it is possible to synchronize all file changes in a directory on a host in real time to a specified location on another host, which can also be achieved by using inotifywait with the rsync command in the script, such as the following script:

#!/bin/bash

desthost=172.16.100.6

desthostdir=/www/htdocs/

srcdir=/www/htdocs/

INOTIFYWAIT-MR--timefmt '%d/%m/%y%h:%m '--format '%T%w%f ' \

-E Create,delete,modify,attrib $SRCDIR | While read DATE time DIR FILE; Do

$FILECHANGE =${dir}${file}

Rsync-avze ' ssh ' $SRCDIR [email protected]${desthost}:${desthostdir} &>/dev/null && \

echo "at ${time} on ${date}, file $FILECHANGE is backed up via rsync" >>/var/log/filesync.log

Done

It is important to note that the rsync in this script is encrypted by SSH file transfer, so you need to configure the corresponding SSH can be based on the key to authenticate users, so that every time the file synchronization requires the user to manually enter the password.

Of course, if the data transfer does not require encryption, it can also be implemented here by starting the RSYNCD daemon on the directory host.

Third, configure the rsyncd+inotify to achieve real-time file synchronization:

This case realizes the monitoring of all file changes in the specified directory on the master machine and synchronizes the changes to the specified directory of the target host in real time, and the host and related directories are as follows:

SOURCE Host: RHEL5.4 (x86), 172.16.100.1, the file is located in the directory of/www/htdocs;

Target Host: RHEL5.4 (x86), 172.16.100.6, the file is located in the directory of/www/htdocs;

1, set the target host (this example is 172.16.100.6)

In this case, data synchronization is based on the rsync daemon, whose data transfer process is plaintext, so it is only suitable for use in specific scenarios.

1) Install related software:

The target host is the server that receives the files sent by other hosts, so its rsync needs to work as a daemon. Rsync services are typically implemented based on the XINETD management of the Super Daemon, so you need to install RYSNC and xinetd in advance:

# yum-y Install rsync xinetd

2) Provide the configuration file/etc/rsyncd.conf for rsync, similar to the following content:

# section 1:global Settings

UID = nobody

GID = Nobody

Use chroot = no

Max connections = 3

Strict modes = yes

PID file =/var/run/rsyncd.pid

Log file =/var/log/rsyncd.log

# section 2:directory to be synced

[Htdocs]

Path =/www/htdocs

Ignore errors = yes

Read Only = no

Write only = no

Hosts allow = 172.16.0.0/16

Hosts deny = *

List = False

UID = root

GID = root

Auth users = Wwwuser

Secrets file =/etc/rsync.passwd

The relevant directives and their descriptions can be obtained through the RSYNCD.CONF manual. The access control function can also be based on xinetd, please refer to Xinetd.conf's manual page for specific methods.

3) Provide the password file/etc/rsync.passwd specified by the secrets file, which resembles the following:

Htdocsuser:passwordforhtdocs

Where the colon before the user name, after the colon is the corresponding user's password. This file cannot be accessed by any other user, so it can be modified using the following command:

# chmod 600/ETC/RSYNC.PASSWD

4) Configuration service can boot up:

# Chkconfig Rsync on

# Chkconfig xinetd on

# service XINETD Start

By default, RSYNCD listens on a port of 873/tcp, which can be viewed with the following command:

# NETSTAT-TNLP | grep ": 873"

TCP 0 0 0.0.0.0:873 0.0.0.0:* LISTEN 3653/xinetd

2. Set the source host

1) Install related software

The source host needs to monitor the events associated with file changes on all files in the specified directory in real time and synchronize the changed data to the directory host when the event occurs, so the source host needs to ensure that the kernel supports inotify and installs Inotify-tools and rsync.

The installation of rsync is done in the same way as the target host.

The installation of Inotify-tools can be done based on source code compilation, or by installing its RPM.

Source: http://github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

RPM Package download page: http://rpm.pbone.net/index.php3/stat/4/idpl/15265939/dir/redhat_el_5/com/inotify-tools-3.14-1.el5.i386.rpm.html

Here's how the installation process is demonstrated by compiling the source code:

# Tar XF inotify-tools-3.14.tar.gz

# CD inotify-tools-3.14

#./configure

# make

# make Install

# echo "/usr/local/lib" >/etc/ld.so.conf.d/usr_local_lib.conf

# Ldconfig

2) provides password files for the source host's rsync to synchronize files to the target host via Htdocsuser users

# echo ' Passwordforhtdocs ' >/etc/rsync.passwd

# chmod 600/ETC/RSYNC.PASSWD

Note that for the RHEL5.4 2.6.8 version of rsync, the password file for the client (the rsync on the source host here) can only save the user's password, not similar to the target host on which the user name can be specified.

3) Establish the script/root/bin/htdocsync.sh, monitor the corresponding event on the target file through inotifywait, and initiate the synchronization process when the event is triggered:

#!/bin/bash

#

desthost=172.16.100.6

desthostdir=/www/htdocs/

srcdir=/www/htdocs/

INOTIFYWAIT-MR--timefmt '%d/%m/%y%h:%m '--format '%T%w%f '-E close_write,modify,delete,create,attrib $SRCDIR | While read DATE time DIR FILE; Do

Filechange=${dir}${file}

Rsync-avz--password-file=/etc/rsync.passwd $SRCDIR [email protected]${desthost}::htdocs &>/dev/null & & \

echo "at ${time} on ${date}, file $FILECHANGE is backed up via rsync" >>/var/log/websync.log

Done

Then execute the permissions on the script and execute it:

# chmod u+x/root/bin/htdocsync.sh

#/root/bin/htdocsync.sh &

If you want this feature to start automatically when you turn it on, you can do so in the following ways:

# echo '/root/bin/htdocsync.sh & ' >>/etc/rc.d/rc.local

Note: If you wish to synchronize the data on the source host to multiple target hosts, set the setting method for each target host similar to the above target host.

This blog post was created by Marco two years ago and shared with you:

Http://blog.chinaunix.net/uid-233544-id-3129307.html

This article from "Marco Education Linux operation and maintenance Training" blog, reproduced please contact the author!

"Linux Tutorial" INotify usage Introduction and combining rsync to achieve real-time file synchronization between hosts