Distributed locks implemented based on RedisLua scripts

A Distributed Lock is needed in recent projects. Considering that the performance of the zookeeper Lock Based on session nodes is insufficient, we want to use redis to implement a distributed lock. After reading several implementation solutions on the internet, I found that they were not rigorous enough. For example, the biggest problem in implementing the locks in Distributed locks using Redis is that the lock timeout value TTL will always be

A Distributed Lock is needed in recent projects. Considering that the performance of the zookeeper Lock Based on session nodes is insufficient, we want to use redis to implement a distributed lock. After reading several implementation solutions on the internet, I found that they were not rigorous enough. For example, the biggest problem in implementing the locks in Distributed locks using Redis is that the lock timeout value TTL will always be

A Distributed Lock is needed in recent projects. Considering that the performance of the zookeeper Lock Based on session nodes is insufficient, we want to use redis to implement a distributed lock. After reading several implementation solutions on the internet, I found that they were not rigorous enough. For example, the biggest problem with implementing the locks in Distributed locks using Redis is that the lock's timeout value TTL will be rewritten all the time. "Although C3 didn't get the lock, however, it overrides the lock timeout value set by C4, but the impact of this tiny error is negligible ", in fact, high concurrency will cause the process to "Starve" (also known as a deadlock ). In this article, v2 = getset (key, time token + timeout + 1) in solution 2 of "Two Distributed Lock implementations ), the addition of 1 second operations in high concurrency will also trigger the same problem. This article on the internet solves this "Endless TTL" problem. I simply translated it.

Lock is a very common concept in programming. On Wikipedia, there is a fairly precise definition of locks:

In computer science, a lock is a synchronization mechanism used to forcibly restrict resource access in a multi-threaded environment. The lock is designed to execute a mutex concurrency control policy.

In computer science, a lock is a synchronization mechanic for enforcing limits on access to a resource in an environment where there are running threads of execution. A lock is designed to enforce a mutual exclusion concurrency control policy.

In short, a lock is a single reference point. Multiple Threads check whether resources are allowed to be accessed based on it. For example, a thread that wants to write data must first check whether a write lock exists. If a write lock exists, wait until the lock is released before it can obtain the lock that belongs to it and perform the write operation. In this way, the lock can avoid data conflicts caused by simultaneous Writing of multiple threads.

Modern Operating systems provide built-in functions to help programmers implement concurrency control, such as flock functions. But what if the multi-thread program runs on multiple machines? How can we control resource access in a distributed system?

Use a centralized lock Service

First, we need a place where all threads can access to store the lock. This lock can only exist in one place, so that only one authoritative place can define the establishment and release of the lock.

Redis is an ideal candidate solution for implementing locks. As a lightweight memory database, fast, transactional, and consistent are the main reasons for choosing redis as the lock service.

Design lock

The lock itself is very simple, it is a simple key in the redis database. Creating and Releasing locks and ensuring absolute security are tricky parts of the lock design. There are two potential traps:

1. The application interacts with redis through the network, which means there is a delay between the application issuing command and redis result return. During this period, redis may be running other commands, and the data status in redis may not be what your program expects. What if the thread used to obtain the lock in the program does not conflict with other threads?

2. If the program suddenly crash after obtaining the lock and cannot release it? This lock will always exist, causing the program to enter the "starved to death" (the original article becomes "deadlock", and it does not feel accurate ).

Create a lock

The simplest way to think of it is to "use the GET method to check the lock. If the lock does not exist, use the SET method to SET a value ".

Although this method is simple, it cannot guarantee an exclusive lock. Review the 1st traps mentioned above: because there is a delay between GET and SET operations, we cannot know whether there are other threads to create locks during the period from "sending command" to "redis server return result. Of course, these are within several milliseconds, and the probability of occurrence is quite low. However, if a large number of concurrent threads and commands are run in a busy environment, the possibility of overlap is not negligible.

To solve this problem, use the SETNX command. SETNX eliminates the need to wait for the returned value of the GET command. SETNX returns success only when the key does not exist. This means that only one thread can successfully run the SETNX command, and other threads will fail, and then retry until they can establish a lock.

Release lock

Once the thread successfully executes the SETNX command, it establishes a lock and can work based on resources. After the work is completed, the thread needs to delete the redis key to release the lock, so that other threads can obtain the lock as soon as possible.

Even so, you also need to be careful! Review the 2nd traps mentioned above: If the thread crash is used, it will never delete the redis key, so this lock will always exist, leading to the "starvation" phenomenon. So how can we avoid this problem?

Lock survival time

We can add a TTL to the lock so that the key of the lock will be automatically deleted by redis once the TTL times out. Any lock left over due to a thread error will be released after a proper time, thus avoiding starvation ". This is purely a security feature, and the more effective way is to ensure that the lock is released within the thread as much as possible.

You can use the PEXPIRE command to set ttl for the Redis key. In the thread, you can run the MULTI/EXEC transaction method immediately after the SETNX command. For example:

MULTISETNX lock-keyPEXPIRE 10000 lock-keyEXEC

However, this may cause another problem. The PEXPIRE command does not determine the returned result of the SETNX command. In any case, the key TTL is set. If the lock cannot be obtained in this place or there is an exception, the TTL of the key will be updated frequently every time multiple threads want to obtain the lock, which will prolong the TTL of the key, the key will never expire. To solve this problem, we need Redis to process this logic in a command. We can use the Redis script.

Note: if you do not use a script, you can use the PX and nx parameters of the set command after Redis 2.6.12. To ensure compatibility with versions earlier than 2.6.0, we still adopt scripts.

Redis script

Because Redis supports scripts, we can write a Lua script to run multiple Redis commands on the Redis server. The application can use an EVALSHA command to call scripts cached by the Redis server. The strength here is that your program only needs to run one command (SCRIPT) to run multiple redis commands in transaction mode and avoid concurrency conflicts, because a redis script can only run once at the same time.

This is a Lua script for Redis to set a lock with TTL:

---- Set a lock---- KEYS[1]   - key-- KEYS[2]   - ttl in ms-- KEYS[3]   - lock contentlocal key     = KEYS[1]local ttl     = KEYS[2]local content = KEYS[3]local lockSet = redis.call('setnx', key, content)if lockSet == 1 then  redis.call('pexpire', key, ttl)endreturn lockSet

From this script, we can clearly see that by running only the PEXPIRE command on the lock, we solved the "Endless TTL" problem mentioned above.

Warlock: A mature Redis-based Distributed Lock

The above describes the Redis-based lock theory. Here is an open-source module Warlock written in Node. js, which can be obtained through npm. It uses the redis script to create/release locks to provide distributed lock services for caching, databases, task queues, and other places that require concurrency. For details, see Github.

There is still a script to release the lock in the original article. If the lock is always released by TTL, the efficiency will be very low. The Redis SET operation documentation provides a script to release the lock:

if redis.call("get", KEYS[1]) == ARGV[1]then    return redis.call("del", KEYS[1])else    return 0end

In the application, you only need to specify a random number or a specific value as the key value when locking. You can use this value to unlock the key. Of course, the value of each lock must be unique.

Original article address: distributed locks Based on Redis Lua script. Thank you for sharing them.