Data consistency check of distributed database cluster node

A 500 strong customer to go online a function, the background of all the database is our design and development of the NoSQL database.

In order to avoid the problem of inconsistent data nodes in the database cluster, it is necessary to verify the data between the database nodes.

In theory, the data between database nodes should be consistent in the final sense. And our database, is the main node in the operation of the data, the Coord node will (immediately) notify the standby node pull data, so as to maintain data consistency. So, for a normal running database, the data on each node in a cluster is exactly the same.

Customer is God, what we do is to reassure customers. While we emphasize that the data in the nodes in our DB cluster is consistent, the only way to reassure customers is to let them see the actual data.

In response to the customer's request, I did a bit of analysis:

In the beginning, I wanted to write a dump tool, dump the database storage file under each node, iterate over each piece of data on each node, and list the different data. The advantages of this method are fast speed and high efficiency. But my thoughts are naïve, my starting point in the beginning, the nodes in the cluster are created at the same time, so that the offsets for each record in the database are consistent. I did not take into account the situation after the expansion.

In the implementers, it is possible to expand at any time, increase the data node, so that the new node and the old node, the initial data, and record the offset may be inconsistent. The new node is created successfully, will actively go to the master node to synchronize, pull the data, while the master node for data operations, will notify the new node synchronization data, I work together, may affect the data record offset, resulting in a new data record offset and the primary node record offset inconsistent. Although the cataloging node also records the offset of data insertion, it can affect the efficiency when the cataloging nodes are queried and then the data records are compared.

So, I chose another way:

Using a C + + driver, each data node's data is queried in the OID ascending order, comparing each record. If there is inconsistent data, record the data first, and record which nodes have that data, and which nodes do not exist for that record. The node cursor that records the minimum OID is then moved down, and cursors of multiple nodes may be moved. Because the OID is the smallest node, moving down one is the next record that is most likely to match. Follow this method to move the cursor gradually to the end of the cursor.

It is possible that when the program is running, the data between the nodes is synchronizing so that the first check results may have inconsistent data. Therefore, you can specify the number of cycles to check, the second check, will not go through all the data, but from the results after the first traversal to select the recorded data, the search for each node, all exist or do not exist to remove the change record.

Theory has, it is practice.

After two days of meditation, the two-day story document was written and endorsed by the leadership. After a week of development and commissioning, the tool has finally come into existence.

I checked the database cluster on my own machine and found no inconsistent data. Our database is still tested:)

Operation of the database node, not through the coord node (thus, from the node can not receive synchronization signal, also do not query to the COORD node data operation), the node for data deletion and modification, and then use the tool test. The result is still what I expected.

That's great! A good sense of accomplishment said:)

The database will open source, and you'll see the tools and code I wrote later:)