Detailed interpretation of file system events-inotify + rsync needs to filter events!

For the use of inotify, an article on IBM is very detailed. I do not need to repeat it here. The link is as follows:

Http://www.ibm.com/developerworks/cn/linux/l-inotifynew/

We are often familiar with inotify-tool, but few people summarize some of its file system monitoring features. Some people use the open source inotify script program for rsync synchronization, but I still want to filter file events. Otherwise, a large number of repetitive and redundant operations will be performed, mainly because, A large number of redundant events are generated when our team files are edited and operated. Some of these events are temporary files and some are redundant operations on the monitoring files. For details, see the test below.

Test environment:

Ubuntu File System ext4

Centos File System ext3

Test Tool: A monitoring program written by inotify API, and prints the event as follows:

Events 256 and 8 are generated, which respectively represent the create event and write_close event.

2. Use the VI statement to create the test file VI test. The event is as follows:

Some temporary files will be generated. They end with SWP and swpx. Among them, 512 events represent the deletion of files. We can see that there are a lot of additional operation overhead behind the VI. However, it is unclear what requires such a complicated process.

3. Write the files opened through VI and execute W at the VI command prompt to generate the following eight events:

This time, some events named 4913 will be generated. This number does not seem to change, that is, you write other files, and the 4913 event will also be generated. However, if the name of the created file is 4913, a temporary file with other numbers will be generated. This is really strange...

Among them, 64 is the move_from event, which is an event generated when the file MV is output to the current path. 128 indicates that the files in other paths are moved to the current path. The cookie value can be used for the outbound and inbound operations, to determine whether the file is the same. It can be seen that when the mobile operation is performed, the test is moved to test ~, Actually, the name is modified. We can see from the cookie that they operate on the same file.

If you exit the file opened by VI, run the Q command:

When the write and close operations are performed on the same file, that is, running WQ under the VI command line generates 10 events as follows:

It can be seen that if the event is not monitored, a simple write operation will generate many redundant events.

However, some events are unique, for example:

1. re-run the touch operation on the generated files. For example, if test has been generated, re-execute the touch test operation as follows:

Only the write_close event is generated, and the touch file is overwritten.

2. From the other path CP to the monitoring path:

It is equivalent to a write overwrite operation.

3. Move a file from another path to the monitoring path:

Only the move_to event is generated.

4. As you can see just now, if the write name is the test file, a temporary file 4913 is generated. If I write a file named 4913, the event is as follows:

If the write value is 5036, other temporary files will be generated, which is really interesting.

Therefore, for those users who want to use inotify and rsync for synchronization, the filtering function should be added to the script.

In subsequent articles, I mentioned my own server synchronization program, which has the following functions:

1. Filter temporary files and unwanted events. For example, only one event is generated during write, so that rsync is required once.

2. When rsync fails, it will be re-executed later. If it still fails, it will be re-executed in 10 hours to ensure the server synchronization accuracy.

3. for files that do not exist locally, I will also delete the remote file path without comparing other files, which use rsync-include =/*-include = xxx. PHP $-exclude = * Users noticed that the execution method is very inefficient and Will recursively compare all directories.

4. supports synchronization from the host to multiple extensions. Do not use multiple threads for simultaneous execution, so that files are consistent across all servers. And synchronize multiple files generated by inotiy at the same time.

5. The number of threads can be configured based on the server performance, so that the server can open more threads and execute more threads at the same time.

Executable files and Source Code address: http://code.google.com/p/sersync/