Research on implementation of distributed lock for MongoDB cluster

The role of distributed locks

In general, the function of a lock is to solve the problem that arises from the competition between different execution flows for the same resource. The function of distributed lock is to solve the problem of the competition of resources in the distributed program, which is distributed on different machines. On MongoDB's cluster, multiple MONGOs will initiate balance this process. And within a period, there can be only one balance process. Therefore, how to resolve multiple MONGOs processes to initiate balance this process requires a distributed lock.

Code location

The code version uses the 2.4.11 version, which is closer to the stable version of our online operation.

The main code files involved are:

Mongo/client/distlock.cpp

Mongo/s/type_locks.h

Mongo/s/type_lockpings.h

Locked Entities

A distributed lock must have a location where the lock is stored. Whether a file lock or a mutex in the kernel or a spin lock, a read-write lock, there is a lock "entity", through the Code of the "entity" to carry out a number of protective operations, to achieve the lock or unlock function.

MongoDB uses the "entity" of the database as the lock. There is an important database config in MongoDB sharding cluster,

There are two collection and distributed locks in this db.

One is config.locks.

Mongos> Db.locks.find (). Pretty () {"_id": "Configupgrade", "process": "i-qikzt805:50000:1390191129:18042 89383 "," state ": 0," ts ": ObjectId (" 52dca219420e5e3bb3e63ee9 ")," when ": Isodate (" 2014-01-20t04:12: 09.751Z ")," who ":" i-qikzt805:50000:1390191129:1804289383:mongosmain:846930886 "," Why ":" Upgrading config Database to new format v4 "} {" _id ":" Balancer "," process ":" qc24:50000:1399171433:1804289383 "," STA TE ": 2," ts ": ObjectId (" 54115f46274b8459f178c927 ")," when ": Isodate (" 2014-09-11t08:37:26.462z ")," Who ":" qc24:50000:1399171433:1804289383:balancer:846930886 "," why ":" Doing balance Round "} {" _id ":" User_ Data.user_data "," process ":" qc-clouddb1:30001:1409913195:236929073 "," state ": 0," ts ": ObjectId (" 5409c74dc3a03d987a4a2d88 ")," when ": Isodate (" 2014-09-05t14:23:09.190z ")," who ":" qc-clouddb1:30001:1409913 195:236929073:conn40:1485371859 "," Why ":" migrate-{_id: \ "824cb5db-6cbd-c90c-95ad-915bd7880c31\"} "} {" _id ":" User_data.device_in Fo "," process ":" qc24:30005:1399282507:1672775172 "," state ": 0," ts ": ObjectId (" 5406f9e738d2c06115 a4e475 ")," when ": Isodate (" 2014-09-03t11:22:15.837z ")," who ":" qc24:30005:1399282507:1672775172:conn149855 6:2012981408 "," Why ":" migrate-{_id: \ "40acae65-45de-ce42-daf6-63fe1b0e8052\"} "}mongos>

You can see that there are 4 locks in existence, "_id": "Configupgrade" as the name implies. "_id": "Balancer" is the lock of the entire balancer process. "_id": "User_data.user_data" and "_id": "User_data.device_info" corresponds to the sharding lock for collection balance.

The other is config.lockpings.

Mongos> Db.lockpings.find (). Pretty () {"_id": "i-qikzt805:50000:1390191129:1804289383", "ping": isodate ( "2014-01-20t09:26:20.903z")} {"_id": "qc14:50000:1398961193:1804289383", "ping": isodate ("2014-09-11t08:41        : 07.546Z "}} {" _id ":" qc24:50000:1399171433:1804289383 "," ping ": Isodate (" 2014-09-11t08:40:56.833z ")} { "_id": "qc23:50000:1399172957:1804289383", "ping": Isodate ("2014-09-11t08:40:55.083z")} {"_id": "QC15: 50000:1399173835:1804289383 "," ping ": Isodate (" 2014-09-11t08:40:54.947z ")} {" _id ":" qc16:50000:1399174043:        1804289383 "," ping ": Isodate (" 2014-09-11t08:41:04.574z ")} {" _id ":" qc24:30005:1399282507:1672775172 ", "Ping": Isodate ("2014-09-05t08:50:18.879z")} {"_id": "qc-clouddb6:50000:1409730027:1804289383", "ping" : Isodate ("2014-09-11t08:40:54.966z")} {"_id": "qc-clouddb7:50000:1409730657:1804289383", "ping": isodate ( "2014-09-11t08:40:54.868Z ")} {" _id ":" qc-clouddb8:50000:1409730659:1804289383 "," ping ": Isodate (" 2014-09-11t08:40:56.802z ")} {  "_id": "qc-clouddb8:30008:1409813212:1448386028", "ping": Isodate ("2014-09-11t08:40:54.989z")} {"_id": "qc-clouddb1:30001:1409913195:236929073", "ping": Isodate ("2014-09-11t08:40:54.947z")} {"_id": "Qc-cloudd b3:30003:1409918540:1296167705 "," ping ": Isodate (" 2014-09-11t08:40:55.232z ")} {" _id ":" Qc-clouddb7:30007:1 409919636:1928209546 "," ping ": Isodate (" 2014-09-11t08:40:54.762z ")} {" _id ":" qc-clouddb2:30002:1409919744:         42373342 "," ping ": Isodate (" 2014-09-11t08:40:52.771z ")} {" _id ":" qc-clouddb6:30006:1409920835:1027944352 ", "Ping": Isodate ("2014-09-11t08:40:54.932z")} {"_id": "qc-clouddb5:30005:1409920983:1186461301", "p ing ": isodate (" 2014-09-11t08:40:54.949z ")} {" _id ":" qc-clouddb4:30004:1409921811:221589655 "," ping ": ISOD Ate ("2014-09-11t08:40:55.083Z ")} 

I-QIKZT805:50000:1390191129:1804289383 is the process ID, which consists of hostname:port:timestamp:random ().

Each MONGOs and Mongod in the cluster will change the ping time of their process ID at intervals. Equivalent to keeping heartbeat.

Lockping will have a single thread to update config.lockpings every once in a while.

Lockpinger Threads

Lockpinger threads Just do 3 things:

1 The time that the update process was in config.lockpings.

2 Check all locks in the config.locks for more than 4 days, and release if they exist.

3 Remove the lock that was not successfully released before the process was removed. (To unlock all locks in the list)

To get a lock

The code to get the lock is mainly in this function distributedlock::lock_try

1 query for the existence of a lock from locks collection (_id== lock name) if it does not exist, you can obtain a lock.

2 If the lock exists, there are 4 situations to discuss:

A. Reentrant, timeout

B. reentrant, no timeout

C. Non-reentrant, timeout

D. Non-reentrant, no timeout

B can get locks, a/c/d can not get locks

It also involves a question of whether to release the lock. C Case will release the lock, otherwise not. And why C is released, and a does not release the reason I did not want to understand. Who can give a finger?

In addition, when the lock is acquired, it is divided into two steps: The first step is to set the state to 1, and if successful, set the state to 2.

Criteria for determining the lock timeout

The last ping arrival time for the lock's owner process has been more than 15 minutes

Steps to unlock

The unlocking step is simple:

1 whether a query lock exists

2 Change Status to 2 if present

3 retry n times after failure, if unsuccessful, put in the unlocked list

Research on implementation of distributed lock for MongoDB cluster