Discussion on the coordination application of Redis in Distributed system _redis

In a distributed system, each process (this article uses a process to describe the running body in a distributed system, they can be on the same physical node also on the different physical nodes are usually required to coordinate with each other, sometimes the data processed by different processes have dependencies, must be processed in a certain order, Sometimes it takes a process to handle some transactions at a certain time, and so on, people often use techniques such as distributed locks and election algorithms to coordinate the behavior between processes. Because of the complex nature of the distributed system itself and the requirements for fault tolerance, these techniques are usually heavyweight, such as the Paxos algorithm, the bullying algorithm, the zookeeper, and so on, which focus on message communication rather than shared memory, and are often known to be complex and incomprehensible, It is a challenge when problems are encountered in specific implementations and implementations.
Redis is often regarded as a kind of NoSQL software, but it is essentially a distributed data structure Server software, which provides a distributed data structure storage service based on memory. On implementation, only one thread is used to handle the specific memory data structure, to guarantee the atomic characteristics of its data manipulation commands, it also supports LUA based scripts, each Redis instance uses the same LUA interpreter to interpret the running Lua script, and thus the Lua script has atomic characteristics, This characteristic of atomic operation makes the coordination of distributed system based on shared memory mode become possible, and with a great deal of appeal, unlike a complex message based mechanism, a pattern based on shared memory is much easier to understand for many technicians, especially those who already understand multithreading or multiple process technologies. In concrete practice, also not all distributed systems like distributed database systems need strict models, and the technology used is not necessarily all need to have a solid theoretical foundation and mathematical proof, which makes the distributed system based on Redis to achieve the coordination technology has a certain practical value, in fact, There have been a lot of attempts. In this paper, some of these coordination techniques are introduced.

Signal/wait operation
in a distributed system, some processes need to wait for the state of other processes to change, or to notify other processes of their own state changes, for example, when there is an operational dependency order between processes, there are processes that need to wait, and processes need to send signals to notify the waiting process for subsequent operations, which can be done through Redis's Pub/sub Series command to complete, such as:

Copy Code code as follows:

Import Redis, Time
rc = Redis. Redis ()
def wait (wait_for):
PS = Rc.pubsub ()
Ps.subscribe (wait_for)
Ps.get_message ()
Wait_msg = None
While True:
msg = Ps.get_message ()
If MSG and msg[' type '] = = ' message ':
wait_msg = Msg
Break
Time.sleep (0.001)
Ps.close ()
Return Wait_msgdef
Signal_broadcast (wait_in, data):
Wait_count = Rc.publish (wait_in, data)
Return Wait_count

This method can be easily extended to implement other waiting strategies, such as try Wait,wait Timeout, waiting for multiple signals to wait for all signals or any one signal to return and so on. Because the Redis itself supports message subscriptions based on pattern matching (using the Psubscribe command), the wait signal can also be set in a pattern-matching manner.
Unlike other data operations, subscription messages are instantly perishable, are not saved in memory, are not persisted, and will not be master/slave if the client's connection to the server is disconnected, but the Publish command is synchronized to the slave node when the node is configured So that we can subscribe to a channel at the same time on both master and slave nodes, so that the publisher's message can be received at the same time, even if master fails in the course of use, or if the connection to master fails, we can still get from the slave The node gets the message of the subscription, thus obtaining better robustness. In addition, this approach has an advantage in performance because the data is not written to disk.
In the above method the signal is broadcast, all in wait process will receive a signal, if you want to set the signal to unicast, only one of them to receive the signal, you can use the contract channel name mode to achieve, such as:
Channel name = channel name prefix (channel) + subscriber globally unique ID (myID)
The unique ID can be a UUID or a random number string to ensure that the global uniqueness is available. Use the "PubSub channels channel*" command to obtain all subscriber-subscribed channels before sending the signal, and then send a signal to one of the randomly assigned channels; when waiting, you need to pass your unique ID, merge the channel name prefix and unique ID into a channel name , and then wait like the previous example. Examples are as follows:

Copy Code code as follows:

Import Random
Single_cast_script= "" "
Local channels = Redis.call (' pubsub ', ' channels ', argv[1] ... ' *');
If #channels = = 0
Then
return 0;
End
Local index= Math.mod (Math.floor (Tonumber (argv[2)), #channels) + 1;
Return Redis.call (' Publish ', Channels[index], argv[3]); """
Def wait_single (channel, myID):
return Wait (channel + myID)
Def signal_single (channel, data):
rand_num = Int (random.random () * 65535)
Return Rc.eval (single_cast_script, 0, Channel, str (rand_num), str (data))

Distributed lock Distributed Locks
The realization of distributed lock is a more and more direction that people explore, the official website of Redis has a document to introduce the distributed lock based on Redis, the Redlock algorithm is presented, and the implementation cases of many languages are listed, here is a brief introduction.
The Redlock algorithm looks at three elements that meet distributed locks:
Security: Guarantees mutual exclusion, at most at any time only one client can hold locks
Deadlock free: Even if the client that is currently holding the lock crashes or is separated from the cluster, the other clients will always be able to get the lock.
Fault tolerance: As long as most of the Redis nodes are online, the client can acquire and release the lock.

A simple and straightforward way to implement the lock is to set a key with the live cycle TTL to acquire the lock by using the Set NX command, to release the lock by removing the key, and to ensure that the deadlock is avoided through the survival cycle. However, this approach has a single point of failure risk, and if the Master/slave node is deployed, it may cause security conflicts under certain conditions, such as:

• Client A gets A lock from the master node
• The master node crashed before copying the key to the slave node
• The slave node is promoted to the new master node
• Client B acquired the lock from the new master node, which was actually held by client A, causing two clients in the system to hold the same mutex in the same time period, destroying the security of the mutex.

In the Redlock algorithm, the lock is performed by a command similar to the following:

Copy Code code as follows:

SET resource_name my_random_value NX PX 30000

The my_random_value here is a globally different random number, each client needs to generate this random number and remember it, and it needs to be used to unlock it later.
Unlocking requires a Lua script that does not simply delete the Key, or it may release the lock that someone else holds:

Copy Code code as follows:

If Redis.call ("Get", keys[1]) = = Argv[1] then return
Redis.call ("Del", Keys[1]) else return 0end

The value of this argv[1] is the value of the my_random_value in front of the lock.
If you need better fault tolerance, you can create a cluster of n (n-odd) Redis redundant nodes that are mutually independent and complete, in which case a client obtains the lock and releases the lock in the following algorithm:
Gets the current timestamp timestamp_1, in milliseconds.
With the same Key and random values, in order to obtain the lock from the N nodes, each acquisition lock set a time-out, the time-out period to ensure that the lock is less than the automatic release time on all nodes, so as to avoid a node on a longer time-consuming, usually set relatively short.
The client subtracts the timestamp timestamp_1 in the first step from the current timestamp, calculating the total time to get the lock consumed. The client is considered to have successfully acquired the lock only if the client obtains a lock of more than half of the nodes, and the total time elapsed is less than the lock lifetime.
If a lock is obtained, its survival time is the start preset lock lifetime minus the total time to acquire the lock.
If the client cannot acquire the lock, it should be unlocked at once on all nodes.
If you want to retry, the lock is retrieved after a random delay.
The client that acquired the lock will release the lock and simply unlock it on all nodes.

The Redlock algorithm does not need to ensure that the clocks between the Redis nodes are synchronized (whether physical or logical), which is different from the traditional distributed lock algorithms based on synchronous clocks. The specific details of the Redlock algorithm can be found in the official documentation for Redis, as well as the implementation of the various language versions listed in the documentation.

Election algorithms
in a distributed system, often some transactions are required to be completed within a certain time period by a process, or by a process as a leader to coordinate other processes, this time need to use the election algorithm, the traditional election algorithm has bullying election algorithm (high-handed election algorithm), ring election algorithm, Paxos algorithm, Zab algorithm (zookeeper) and so on, some of these algorithms rely on the reliable transmission of messages and clock synchronization, some too complex, difficult to implement and verify. The new Raft algorithm is much easier compared to other algorithms, but it still relies on heartbeat broadcasts and logical clocks, leader need to keep the follower broadcast messages to maintain dependencies, and other algorithms are needed to extend the nodes.
Electoral algorithms and distributed locks are somewhat similar, with at most one leader resource at any given time. Of course, we can also use the distributed lock described above, set up a leader resource, get this resource lock for leader, lock life cycle after, and then compete for this resource lock. This is a competitive algorithm, this method will lead to a relatively large gap period without leader, and not to achieve leader re-election, and leader re-election is a relatively large advantage, such as leader to perform tasks can be more punctual, It is also convenient to view logs and troubleshoot problems, if we need an algorithm to achieve leader can be re-elected, then you can use this method:

Copy Code code as follows:

Import Redis
rc = Redis. Redis ()
Local_selector = 0def Master ():
Global Local_selector
Master_selector = RC.INCR (' Master_selector ')
If Master_selector = 1:
# initial/restarted
Local_selector = Master_selector
Else
If Local_selector > 0: # I ' m the master before
If Local_selector > Master_selector: # Lost, maybe the db is fail-overed.
Local_selector = 0
Else: # continue to being the master
Local_selector = Master_selector
If Local_selector > 0: # I ' m the current master
Rc.expire (' Master_selector ',) return local_selector > 0

This algorithm encourages re-election, only if the current leader fails or takes more time to perform a task than the term, or if the Redis node fails to recover, the new leader will need to be re-elected. In Master/slave mode, if the master node fails, a Slave node is promoted to the new master node, and even if the Master_selector value is not synchronized successfully, it will not cause two leader to happen. If a leader has been re-elected, the value of the master_selector will continue to increase, considering that Master_selector is a 64-bit integer type that is not possible to overflow in the foreseeable time, plus every time the leader is replaced, master _selector will be reset to 1, this incremental approach is acceptable, but it is necessary to deal with Redis clients (such as node.js) that do not support 64-bit integer types. If the current leader process exceeds the term, the other processes can regenerate the new leader process, and after the old leader process finishes processing the transaction, if the new leader process experiences more than or equal to the number of terms of the old leader process, There may be two leader processes, and in order to avoid this situation, each leader process should check whether its processing time exceeds the term of office, and if it exceeds the term, you should set the Local_selector to 0 before calling Master Check if you are a leader process.

Message Queuing
Message Queuing is the basic infrastructure of communication between distributed systems, which can construct complex process coordination operations and interoperation through messages. Redis also provides primitives for constructing message queues, such as the Pub/sub Series command, which provides a way to send and receive messages based on the subscription/publish pattern, but pub/sub messages are not persisted within Redis and thus are not persistent. A scenario that applies to a message that is transmitted, even if it is lost.
If you want to consider persistence, you can consider the list series operation commands, push series commands (Lpush, Rpush, etc.) to send messages to a list, use the POP Series command (Lpop, Rpop,blpop,brpop, etc.) to get the message on a list, by not The same combination of ways can get Fifo,filo, for example:

Copy Code code as follows:

Import Redis
rc = Redis. Redis ()
def fifo_push (q, data):
Rc.lpush (q, data)
def fifo_pop (q):
return Rc.rpop (q)
def filo_push (q, data):
Rc.lpush (q, data)
def filo_pop (q):
return Rc.lpop (q)

If you replace Lpop with the Blpop,brpop command, Rpop also supports blocking waiting when the list is empty. However, even if the persistence is implemented in this way, if a network failure occurs when the POP message returns, a message loss is still occurring, and for this requirement Redis provides Rpoplpush and Brpoplpush commands to first save the extracted message in another list. The client can view and process the message data from this list first, and then remove the message data from the list after processing, ensuring that the message is not lost, as the following example:

Copy Code code as follows:

def safe_fifo_push (q, data):
Rc.lpush (q, data)
def safe_fifo_pop (q, Cache):
msg = Rc.rpoplpush (q, cache) # Check and do something on MSG
Rc.lrem (Cache, 1) # Remove the msg in cache list. Return msg

If you use the Brpoplpush command instead of the Rpoplpush command, you can block the wait while the Q is empty.