In-depth research on data replication in Linux

In-depth research on data replication in Linux-Linux Enterprise Application-Linux server application information. The following is a detailed description. In this article, we will discuss how to replace the expensive SAN (Storage Area Network (such as GFS) or Network block device for Data Replication in Linux. We used FAM (file change monitoring module) and IMON (information node monitoring module) in the replication system. Both systems were originally developed for IRIX by SGI.

SGI employees are really Cool. They transplanted the two programs to Linux and published the source code. When the cost is no longer a problem, you can use GFS (Global File System) and SAN to implement real-time data mirroring; otherwise, data sharing and many other options are indispensable.

There are several methods to choose from. In this article, we will discuss these methods, and you will see their respective advantages and disadvantages.

Why replace sharing with replication?

Is it assumed that the file server does not provide shared data to the client? Yes, it is assumed that the work environment is indeed like this. If the file server we are using is shared files through software such as NFS or SMB, there will be a "bottleneck" and "Key Points causing system faults" in the system ". If shared data is shared through shared storage devices (SAN or multi-channel SCSI) on GFS, this configuration is not only expensive, but also becomes the "Key to system failure ". You can also use NBD (network block device) to create a network image, but this is not a common method. NBD itself has some limitations and is very difficult to set and manage, if you only want to copy data between a few WEB servers, using NBD will only bring you more trouble.

As simple as possible

Okay. Let's try copying it.

Solution 1:

One of the two WEB servers is the master server and the other is the backup server. Files on the backup server must be the same as those on the master server. This is simple.

But how can we make it work automatically? The user replicates data to the master server through FTP multiple times a day. But what happens when an error occurs on the master server and the backup server takes over the system? Because this backup operation is not real-time, the data on the backup server must be inconsistent with that on the master server. In this case, the Administrator will be very angry. Of course, you can run a timer daemon every 5 seconds. "rsync? Av? Delete source destination ", but this will increase the load on the machine and affect the operation of the system.

Solution 2:

One FTP server is used to save and update web data, and the six web servers use dns rotation to achieve load balancing. This ensures that the data on each server is the same. In this way, we can avoid using NFS, but such a solution is not satisfactory.

So what is the best solution? It should be "copy the file to each web server only when the data changes". If the file does not change, nothing will be done. This is what we use FAM to do.

Make it work intelligently

So, how do we know that the file has changed? I think the Windows developer of M $ will say this: we can search for the target directory every few seconds and compare the files in the directory with the files saved in the cache, check whether the file Timestamp and size have changed. Yes, that's right.

This is a method of polling. The problem is that it is expensive to search for and compare the timestamp and size of a file. You can imagine that the WEB server runs every five seconds. "ls? What are the consequences of lR/somedirectory?

The best way is: someone can tell us when the file is changed, so that we can take appropriate actions. This is exactly what "IMON" is going to do.

What is FAM?

FAM is the file change monitoring module. It provides a set of APIs to applications. When a specified file or directory changes, it sends a notification to the application. FAM consists of two parts: the background daemon FAM, which receives requests and sends notifications. The library file LIBFAM is used by the client application to communicate with FAM. If the monitored file is opened on the remote host, the local FAM will contact the FAM of the remote host and send the request to the remote FAM.

FAM can also notify its client when a file starts or stops running. (For example, in an IRIX interactive desktop, if a program is running, its icon will not flash ).

FAM was written by Bruce Karsh for IRIX in 1989 and rewritten by Bob Miller in 1995. This open-source FAM version is built and runs on Linux and IRIX, which is the same as FAM in IRIX6.5.8.