Slime: rsync and inotify integration for real-time data synchronization updates

This article by show according to Lin Feng to provide friendship sponsorship, starting in the mud row world.

In the previous article we described how to use rsync synchronization files, this article we will introduce, how to integrate rsync and inotify to achieve real-time data synchronization.

To achieve this goal, we need to take the following steps:

1, the advantages and disadvantages of rsync

2. What is INotify?

3. Check if OS supports INotify

4, inotify related parameters detailed

5, inotify Monitoring of the file events similar

6. What is Inotify-tools?

7, Installation Inotify-tools

8, inotifywait use detailed

9, Inotifywatch use detailed

10, Inotif-tools and rsync integration

First, the advantages and disadvantages of rsync

Rsync is a more important and practical service under Linux/unix, and you should already know that rsync has the advantages of high security, fast backup, and support for incremental backup.

Rsync can be used to solve the requirements of low-real-time data backup, such as: regularly back up file server data to the remote server, the local disk to do regular data mirroring.

With the scale of application system expanding, the data security and reliability also put forward higher requirements, rsync in the high-end business system also gradually exposed its shortcomings.

First, when rsync synchronizes data, it needs to scan all the files before doing the comparison, then the difference is transmitted. If the number of files reaches millions or even tens, scanning so the file will be very time consuming. And the change is often a very small part of the file, which is a very inefficient way.

Second, rsync is not able to monitor and synchronize data in real time. Although it can trigger synchronization through the Linux daemon, there must be a time difference between the two triggering actions. This can lead to inconsistent server and client data, and the inability to fully recover data in the event of an application failure.

For these reasons, consider using rsync and inotify integration to solve these problems.

Second, what is INotify

INotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism.

Since 2.6.13 (August 2005), the Linux kernel has added support for inotify, which can be used to monitor the various minor events such as additions, deletions, modifications, and movements in the file system via INotify. Using this kernel interface, third-party software can monitor the file system under the various changes, and inotify-tools is such a third-party software.

In the above section, we mentioned that rsync can implement trigger-type file synchronization. It is triggered by the crontab daemon, synchronized data and actual data will be different, and inotify can monitor the file system changes, when the file has any changes, trigger rsync synchronization, which just solves the problem of real-time data synchronization.

Third, detect whether the OS supports inotify

Since the INotify feature requires the support of the Linux kernel, it is necessary to verify that the Linux kernel has reached 2.6 13 before installing Inotify-tools. If the Linux kernel is below the 2.6.23 version, you will need to recompile the kernel to add support for inotify, or you can use the following method to determine whether the Linux kernel supports inotify.

Note: The OS for this article is currently CentOS 6.5 64bit.

Cat/etc/system-release

Uname-r

ls-lsart/proc/sys/fs/inotify/

As long as the ls-lsart/proc/sys/fs/inotify/command is executed, the following results are Max_user_watches, max_user_instances, max_queued_events three files, indicating that the CentOS 6.5 64bit is supported for inotify.

Through, we can obviously see that the CentOS 6.5 64bit is supported by INotify.

Four, inotify related parameters detailed

INotify defines three interface parameters that can be used to limit the size inotify consumes kernel memory. Because these parameters are memory parameters, they can be resized in real time based on application requirements. Here is a brief introduction:

/proc/sys/fs/inotify/max_queued_evnets

Represents the maximum value of the number of events requested, and events that exceed this value are discarded. The value defaults to 16384.

Note: max_queued_events is the maximum length of the queue that INotify manages, and the more frequently the file system changes, the greater the value should be.

If you see the event Queue Overflow in the log, it means that max_queued_events is too small to be used again after adjusting the parameters.

/proc/sys/fs/inotify/max_user_instances

Represents the maximum number of instances that can be created per user. The value defaults to 128.

/proc/sys/fs/inotify/max_user_watches

Represents the maximum number of directories that can be monitored. The value defaults to 8192.

To modify the above default values, we can use the following similar means to modify. As follows:

Echo 30000000>/proc/sys/fs/inotify/max_user_watches

V. Types of file events monitored by inotify

INotify the types of file system events that can be monitored, as follows:

In_access: file is accessed.

In_modify: File is write.

In_attrib: File attributes are modified, such as chmod, Chown, and so on.

In_close_write: Writable file is CLOSE.

In_close_nowrite: The non-writable file is CLOSE.

In_open: File is OPEN.

In_moved_from: Files are moved out of the monitored directory, such as MV.

In_moved_to: Files are moved into the monitored directory, such as MV, CP.

In_create: File/folder is created.

In_delete: Files/folders are deleted, such as RM.

In_delete_self: Self-deletion, that is, an executable file deletes itself when it is executed.

In_move_self: Self-moving, that is, an executable file moves itself at execution time.

In_unmount: The host file system is umount.

In_close: File is closed, equivalent to (in_close_write| In_close_nowrite).

In_move: File is moved, equivalent to (in_moved_from| IN_MOVED_TO).

Note: The files mentioned above also include directories.

Vi. What is Inotify-tools?

INotify is just an API that needs to be called by developing an application. Inotify-tools is such a inotify software, it is a set of components, including a C library and a few command-line tools. These command-line tools can be used to monitor the events of certain file systems through the command line or script.

INotify is designed to replace Dnotify, which overcomes the shortcomings of dnotify and provides a better, more concise and powerful notification mechanism for file changes.

1) inotify does not need to open the file descriptor to the target being monitored, and if the target is being monitored on removable media, the watch corresponding to the monitored target will be automatically deleted after umount the file system on that media, and a Umount event will be generated.

2) INotify can monitor both files and directories.

3) INotify uses system calls rather than Sigio signals to notify file system events.

4) inotify uses a file descriptor as an interface, so you can use the usual file I/O operations Select and poll to monitor file system changes.

Seven, Installing Inotify-tools

The installation of Inotify-tools can be divided into source mode and RPM mode. Here are one by one explanations of these two ways.

Note: Inotify-tools is mainly done by inotifywait and Inotifywatch, both commands. In particular, the inotifywait command is the most used command in our production environment.

7.1 Source Mode installation

Source mode installation Inotify-tools, we can go to inotify-tools official website to download the source package.

Inotify-tools Official website:

Https://github.com/rvoicilas/inotify-tools/wiki

Download the Inotify-tools and install it as follows:

Note: It is not recommended to use wget to download inotify-tools, because I have been unsuccessful in using wget download Inotify-tools, and later query related information found to be the reason for GitHub website.

wget https://cloud.github.com/downloads/rvoicilas/inotify-tools/inotify-tools-3.14.tar.gz

We can first download the local through the browser, and then through the RZ command to upload to the server.

To use the RZ command, we need to install the LRZSZ software as follows:

Yum-y Install Lrzsz

Lrzsz after installation, we will upload the Inotify-tools package. As follows:

Now start extracting and installing Inotify-tools, as follows:

TAR-XF inotify-tools-3.14.tar.gz

./configure

Make&&make Install

After the installation is complete, we switch to the/usr/local/bin/directory to view, as follows:

cd/usr/local/bin/

Through, we can see that the inotifywait and Inotifywatch commands have been installed into the/usr/local/bin/directory.

7.2 RPM Mode installation

To install Inotify-tools in RPM mode, we first configure the Yum source, otherwise the system will not be prompted to find the Inotify-tools package. As follows:

We can go to the following link to download the latest Yum source, as follows:

http://dl.fedoraproject.org/pub/epel/6/x86_64/

Download the epel-release-6-8.noarch.rpm package and install it as follows:

wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

RPM-IVH epel-release-6-8.noarch.rpm

After the above operation, we come to Yum to install Inotify-tools, as follows:

Yum–y Install Inotify-tools

View the generated files for the Inotify-tools installation as follows:

RPM-QL Inotify-tools

Through, we can easily see that inotifywait and Inotifywatch have been installed in the/usr/bin/directory.

The above is the installation of the Inotify-tools.

Viii. inotifywait use of the detailed

Inotify-tools command, the most we use is the inotifywait command.

Inotifywait is a monitoring wait event that is primarily used to monitor changes to a file or directory and to recursively monitor the entire directory tree.

8.1 inotifywait Command Explanation

For information on how to use inotifywait, we can view it through its help. As follows:

Inotifywait-h

Through, we can see a lot of inotifywait parameters. Here we introduce some of the frequently used parameters:

-M indicates that the event listening state is always maintained.

-R indicates a recursive query directory.

-Q indicates that a monitoring event is printed.

-e This parameter allows you to specify which events to monitor. The events that can be monitored are as follows:

Access: accessing, reading files.

Modify: Modify, the contents of the file are modified.

Attrib: property, file metadata is modified.

Move: Moves and moves the file. Rename the file and rename it.

Create: Creating, generating new files

Open: Opens to open the file.

Close: Closes and closes the file.

Delete: deleted, the file is deleted.

--TIMEFMT is the output format for the specified time and is used for the%t format in the--format option.

--FORMAT Specifies the detail output format for file changes. The format parameters are as follows:

%w indicates the directory where the event occurred

%f indicates the file in which the event occurred

%e indicates that an event occurred

%t using the time format defined by--TIMEFMT

%xe events are separated by "X"

8.2 inotifywait Command Instance

8.1 Chapters we explain the parameters of the inotifywait command, and let's actually use the next inotifywait command.

We want to monitor the changes of all files and directories under the/home directory, and command the following:

Inotifywait-mrq--timefmt '%y/%m/%d/%h:%m '--format '%T%w%f%e '-e modify,delete,create,attrib,move,open,close,access /home/

This command indicates that the operation of all files and directories under the/home directory is monitored.

Now let's open another window to manipulate the/home directory as follows:

We created an empty file CreateFile and a new directory Createmkdir under the/home directory.

Now let's switch to the window that just executed the inotifywait command, as follows:

Through, we can easily see that inotifywait has been monitoring the files and directories we created earlier.

Nine, Inotifywatch use detailed

In addition to the Inotifywatit command, Inotify-tools also has a command inotifywatch.

Inotifywatch is primarily used to collect monitored file system statistics, including the number of secondary messages per INotify event.

9.1 Inotifywatch Command Explanation

Inotifywatch can be used to view its help information, as follows:

Inotifywatch-h

The inotifywatch parameters are described below:

-H: Output help information.

-V: Output details.

@: Excludes files that do not need to be monitored, either as a relative or an absolute path.

–-fromfile: Reads from files that need to be monitored or excluded files, a single line of files, and excluded files begin with @.

-Z: Outputs the rows and columns of the table, even if the element is empty.

–-exclude: Regular matching files that need to be excluded are case-sensitive.

–-excludei: Regular matches files that need to be excluded, ignoring case.

-r: Monitors all subdirectories under a directory.

-T: Sets the time-out period.

-E: Listens only for the specified event. The event is the same as the type of event that inotifywait listens to.

-A: In ascending order of specified events.

-D: Sorts the specified events in descending order.

9.2 Inotifywatch Command Instance

9.1 Chapters We explain the parameters of the Inotifywatch command, and let's actually use the next Inotifywatch command.

Required to count the file system events in the/home directory within 60 seconds, using the following command:

Inotifywatch-v-E modify,delete,create,attrib,move,open,close,access-e modify-t 60-r/Home

Now let's open another window to manipulate the/home directory as follows:

We are deleting a file in the/home directory CreateFile and renaming Createmkdir to test.

Now let's switch to the window that just executed the Inotifywatch command, as follows:

By, we can easily see that inotifywatch has monitored the number of files and renamed directory file events that we have previously deleted.

X. Integration of Inotify-tools and rsync

The integration of Inotify-tools and rsync is mainly achieved through the integration of the inotifywait command with the rsync command, and the integration is mainly embodied in the Rsync client, and the rsync server needs to follow the normal configuration.

Note: The main purpose of this integration is to push the files that the Rsync client needs to be backed up to the rsync server in real time.

For the configuration of rsync, you can refer to the "Slime: Linux file Synchronization rsync learning (i)" article, and the Environment of this experiment and this article is the same environment.

The rsync server is the 192.168.199.247,rsync client for 192.168.199.248.

Our request now is that as long as there are any files or directories in the/home/www directory of the Rsync client, they should be synchronized to the Rsync server in real time.

The rsync server is working properly with the following configuration files:

All of our operations are done on the rsync client, in fact the inotifywait command integrates with the rsync command, and all we have to do is write a shell script. The script reads as follows:

#!/bin/bash

src=/root/www/

Dest=www

ip=192.168.199.247

/usr/bin/inotifywait-mrq--timefmt '%d/%m/%y%h:%m '--format '%T%w%f '-E modify,delete,create,attrib $src | While read DATE time DIR FILE;

Do

Filechange=${dir}${file}

/usr/bin/rsync-avz--delete--progress $src [email protected] $ip:: $dest--password-file=/etc/rsyncd.password &

echo "at ${time} on ${date}, file $filechange is backed up via rsynce" >>/tmp/rsync.log 2>&1

Done

In this script, I'm only explaining the while read DATE time DIR file statement, and other statements please check the relevant documentation yourself.

The inotifywait command produces three return values, namely "date, time, file", which are passed as parameters to read, so that the "while read D E F" in the script refines the return value.

The principle of this script is to use the inotifywait command to monitor the specified files and directories, and to initiate the Rsync synchronization command if the specified file and directory changes.

After the above script is written, we want to give it permission to execute, then configure its boot background to run and start. Use the following command:

chmod inotify.sh

chmod u+x inotify.sh

ll |grep inotify.sh

echo "sh/root/inotify.sh >/dev/null &" >>/etc/rc.local

SH inotify.sh >/dev/null &

Note: In order for the script to start running in the background, be sure to write the startup command in the form of a diagram, otherwise the script will report the following information when synchronizing the file:

When the above configuration is complete, let's test it to see how it works. Create a new file ilanni.txt on the Rsync client, as follows:

Touch Www/ilanni.txt

Now switch on the rsync server to see if the file you just created on the rsync client has been synced.

Through, we can obviously see that the files have been synced over.

The rsync client then looks at the related logs as follows:

You can see the files in the log that were synchronized in the time that the synchronization was recorded.

The integration of Inotify-tools with Rsync has been fully configured.

Slime: rsync and inotify integration for real-time data synchronization updates