Mailbox: supports performance tuning and cluster migration of the Yixin database on a daily basis

During the rapid expansion of Mailbox, one of the performance problems is the MongoDB database-level write lock. the time consumed during the lock wait process directly reflects the latency during the user's service usage. To solve this long-standing problem, we decided to migrate a common MongoDB collection to an independent cluster. We can infer that this will reduce the lock wait time by 50%. At the same time, we can add more shards, and we expect to optimize and manage different types of data independently.

We started from the MongoDB documentation and soon found the cloneCollection command. However, the tragedy later found that it cannot be used in the fragment set; similarly, renameCollection cannot be used in the fragment set. Based on performance issues after denying other possibilities), we have compiled a Python script to copy data and another script to compare original and target data. In this process, we also found a lot of interesting things, for example, the time for copying a large dataset from gevent and pymongo is half the time for copying a large dataset, even if the MongoDB client and server are on the same host. Through final efforts, we developed Hydra, a tool set for MongoDB migration, which is now open-source. First, we have created the original snapshot of the MongoDB set.

Question 1: Tragic Performance

In the early days, I made an experiment to test the maximum speed that MongoDB APIs can achieve-enabling a simple speed to use MongoDB C ++ software development kit. On the one hand, I am bored with C ++. On the other hand, I hope most of my colleagues who are familiar with Python can use or adapt to this code for other purposes. I have not explored the usage of C ++ further, however, it is found that for a small amount of data, the speed of a simple C ++ application is 5-10 times faster than that of a simple Python application in terms of processing the same task.

So I went back to Python, the default Dropbox language. In addition, when a series of remote network requests such as mongod queries are made, the client usually takes a lot of time to wait for the server to respond. It seems that there are not many copy_collection.py my MongoDB collection replication tools) CPU-intensive operations required ). Initialcopy_collection.py also confirms this.

Then, MongoDB requests copy_collection.py .. The initial working thread experiment results were not ideal. However, we will use the Python Queue object to implement thread communication. This performance is still not very good, because the overhead on IPC has eclipsed the increase in concurrency. Using Pipes and other IPC Mechanisms is not very helpful.

Next, we tried to use a single-thread Python for MongoDB asynchronous query to see how much Performance balance can be achieved. Gevent is one of the common libraries used to implement this approach. We tried it. Gevent modifies the standard Python module to implement asynchronous operations, such as socket. The better thing is that you can write the code asynchronously, just like synchronizing the code.

Def copy_documents (source_collection, destination_collection, _ ids, callback ):

"""

Given a list of _ id's (MongoDB's unique identifier field for each document ),

Copies the corresponding events from the source collection to the destination

Collection

"""

Def _ copy_documents_callback (...):

If error_detected ():

Callback (error)

# Copy documents, passing a callback function that will handle errors and

# Other communications

For _ id in _ ids:

Copy_document (source_collection, destination_collection, _ id,

_ Copy_documents_callback)

Http://www.dianjiare.net

# More error handling omitted for breted

Callback (None)

Def copy_document (source_collection, destination_collection, _ id, callback ):

"""

Copies document corresponding to the given _ id from the source to

Destination.

"""

Def _ insert_doc (doc ):

"Http://www.kongzhixitong.net

Callback that takes the document read from the source collection

And inserts it into destination collection

"""

If error_detected ():

Callback (error)

Destination_collection.insert (doc, callback)

# Another MongoDB operation

# Find the specified document asynchronously, passing a callback to receive

# The retrieved data

Source_collection.find_one ({'$ id': _ id}, callback = _ insert_doc)

With gevent, the Code no longer needs to use callback: http://www.dianjiarequan.net

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

Import gevent

Gevent. monkey. patch_all ()

Def copy_documents (source_collection, destination_collection, _ ids ):

"""

Given a list of _ id's (MongoDB's unique identifier field for each document ),

Copies the corresponding events from the source collection to the destination

Collectio n http://www.dantouguan.net

"""

# Copies each document using a separate greenlet; optimizations are certainly

# Possible but omitted in this example

For _ id in _ ids:

Gevent. spawn (copy_document, source_collection, destination_collection, _ id)

Def copy_document (source_collection, destination_collection, _ id ):

"""

Copies document corresponding to the given _ id from the source to

Destination. http://www.yetijiare.com

"""

# Both of the following function callblock without gevent; with gevent they

# Simply cede control to another greenlet while waiting for Mongo to respond

Source_doc = source_collection.find_one ({'$ id': _ id })

Destination_collection.insert (source_doc)

# Another MongoDB operation

This simple code can copy the code from the MongoDB source set to the target location based on their _ idfields. Their _ idfields are the unique identifier of each MongoDB document. Opy_documents delegates greenlets to run runcopy_document () for document copying. When greenlets performs a blocking operation, such as any requirement for MongoDB, it will control the execution to other prepared greenlet. Because all greenlets are executed in the same thread and thread, you generally do not need any form of internal locking.

Http://www.wujindianqi.net

With gevent, you can find a faster method than the worker thread pool or worker process pool. The performance of each method is summarized below:

ApproachPerformance (higher is better)

Single process, no gevent520 events ents/sec

Thread worker pool652 documents/sec

Process worker pool670 documents ents/sec

Single process, with gevent2, 381 documents/sec

A linear increase in performance can be achieved by combining gevent and worker process with each shard. The key to using working processes effectively is to use less IPC as much as possible.

Question 2: Replication modification http://www.sujiaojixie.net after Snapshot

Because MongoDB does not support transactions, if you read a large dataset that is being modified, the results may vary from time to time. For example, if you use MongoDB find () to read the entire dataset, your result set may be:

Ncluded: document saved before your find ()

Included: document saved before your find ()

Included: document saved before your find ()

Included: document inserted after your find () began

In addition, in order to minimize the fault time when the Mailbox backend points to the new replica set, it is critical to minimize the time spent from the application of the source cluster to the new cluster.

Similar to most asynchronous replication storage, MongoDB uses the operation log oplog to record the addition, modification, and deletion operations on the mongod instance, which is used to allocate all copies to this mongod instance. In view of the snapshot, The oplog records all changes after the snapshot has taken place.

Therefore, the job here becomes to apply the oplog record of the source cluster on the target cluster. From Kristina Chodorow's teaching blog, we understand the oplog format. In view of the serialized format, addition and deletion are very easy to execute, and modification has become one of the difficulties.

The modified oplog logging structure is not very friendly: duplicate keys are used in MongoDB 2.2. However, these duplicate keys cannot be presented through Mongo shell, not to mention most MongoDB drivers. After careful consideration, I chose a simple work und: Embed _ id into the source document to trigger other copies of the document. Because it is only for modification, although the replica set and the source instance cannot be fully synchronized, the gap between the real-time status of the replica set and the snapshot can be minimized. The following figure shows why the intermediate version v2 is not necessarily the same, but the source and target copies are still consistent:

The performance problem of the target cluster also occurs here: although an independent process is used for the ops of each shard, the continuous ops performance still cannot match the requirements of Mailbox. Http://www.mojujiare.com

In this way, ops concurrency becomes a required path, but the correctness guarantee is not easy. In particular, the same _ id operation must be executed sequentially. Here, a Python set is used to maintain the _ id set of the ops being modified: when a request is made on copy_collection.py, the system blocks all the ops applied after modification or other operations until the old operation ends. : Http://www.dianjiarebang.com

>

Verify Data Replication

Comparing replica sets with source instance data is usually a simple operation, but it is a great challenge to do so in multi-process and multi-namespace. At the same time, data is constantly being modified, so there are more things to consider:

First, use compare_collections.py as the tool for data development) to verify the data of the recently modified documents. If there is any inconsistency, a reminder will be sent, and then a review will be conducted. However, the deletion of the document is not valid because there is no last modified timestamp.

Http://www.dianjiareguan.net

Next I think of "final consistency" because it is very popular in asynchronous scenarios, such as MongoDB replica set and MySQL Master/Slave replication. After a lot of attempts, the source data and copies will be eventually consistent except for the next big fault scenario. Therefore, we made some repeated comparisons and continuously increased backoff during continuous retry. Some problems still exist. For example, the data may not change between two values. However, in the modification mode, the migrated data does not have any problems.

Before performing the final conversion of the New and Old MongoDB clusters, ensure that ops has been applied recently. Therefore, we added the command line option in compare_collections.py to compare the last N operations modified in the document, this can effectively avoid inconsistency. This operation does not take too much time. It takes only a few minutes to compare the number of ops executed by a single slice to 100,000, but also relieves the pressure on comparison and retry approaches.

Handling exceptions

Although multiple methods are used to handle error retries, discover possible exceptions, and logs), there are still many unexpected errors in the final test before the product migration. Some irregular network problems have occurred. A specific document Assembly has caused mongos to be disconnected from copy_collection.py and reset from mongod.

After trying, we found that a special solution was developed to address these problems, so we quickly switched to fault recovery. We recorded the doc_id detected by compare_collections.py, and then created a document duplication tool for these _ IDs.

Final migration time

During the product migration process, copy_collection.py created an original snapshot of tens of millions of emails and reproduced over MongoDB ops. The entire replication process lasted for about nine hours by executing the original snapshot and creating the index, and we set the time limit to 24 hours. During this period, we used copy_collection.py to repeat three times and checked the data to be copied three times.

The entire conversion is completed only today, And MongoDB-related work is only a few minutes ). In a concise maintenance window, we use compare_collections.py to compare the last 0.5 million ops of each shard. After ensuring that there is no inconsistency in the final operation, we did some related tests, then directed the Mailbox backend to the new cluster, and re-opened the services for users. After the conversion, we did not receive any user feedback. Migration is the greatest success if you do not feel it. The improvements after migration are shown in:

This article from the "Electronic Design" blog, please be sure to keep this source http://tnjheater.blog.51cto.com/8000139/1304189