Redis Lua script

Background

Redis database provides some management functions such

Assembly line: Package and send multiple commands, and receive the results of all executed commands in one response.
Transaction: when multiple commands are executed at a time, the executed commands are either executed or not executed. And the transaction has been executed
It will not be interrupted by other work in the process.
Optimistic lock: monitors specific keys to prevent transactions from competing.
Although these additional functions are very useful, they also have some defects.

Pipeline Defects
Although the pipeline can be used to send multiple commands at a time, for a complex operation composed of multiple commands
This is not the most efficient method and will waste network resources.
If we can avoid sending the same command repeatedly, the client can reduce the time spent on network transmission and perform operations.
It can be executed faster.

Flaws in transactions and optimistic locks
Although transactions can be used to execute multiple commands at a time, and optimistic locks can be used to prevent the transaction from competing, in practice
Correct use of transactions and optimistic locks is not easy.
1. for a complex transaction, you usually need to think carefully to know which keys should be locked: locking keys that should not be locked will increase the chance of transaction failure, it may even cause a program error. If you forget to lock the key that should be locked, the program will have competition conditions.
2. sometimes, to prevent the occurrence of competitive conditions, even if the operation itself does not need to use transactions, but to make the optimistic lock take effect, we will also use transactions to wrap the commands, which increases the implementation complexity, it also causes extra performance loss.

Misuse examples
The implementation of the zdecrby command is introduced in the transaction section. Here the transaction is only used to make watch take effect:
Def zdecrby (Key, decrment, member ):
# Ordered set of monitoring inputs
Watch key
# Obtain the current score of an element
Old_score = zscore key member
# Use the current score minus the specified reduction to get a new score
New_score = old_score-decrment
# Use the transaction package zadd command
# Make sure that the zadd command is executed only when the ordered set is not modified.
Multi
Zadd key new_score member # sets a new score for the element to overwrite the existing score
Exec

How to avoid misuse of transactions
If there is a method that allows us to execute multiple commands in transaction mode, and this method does not introduce any competition conditions, we can use this method to replace transactions and optimistic locks.

Difficulties in scaling redis Functions
Redis provides corresponding operation commands for each data structure and the database itself. However, if we need to perform operations that are not supported by redis commands on the data structure, you need to use the client to retrieve the data, then the client processes the data, and finally store the processed data back to the redis server.
For example, because redis does not provide the command to delete all the even numbers in the list, the client needs to retrieve all the items in the list to perform this operation, then, filter the items in the client, and push the items to the list again:
LST = lrange lst 0-1 # retrieve all elements contained in the List
Del lst # delete an existing list
For item in lst: # traverse the entire list
If item % 2! = 0: # push non-even elements into the list
Rpush lst item
In addition, to ensure the security of this operation, it is very troublesome to use transactions and optimistic locks.

Lua script
To solve the problem mentioned above, redis has embedded a Lua interpreter on the server since version 2.6, allowing you to execute the Lua script on the server.
This feature has the following benefits:
1. scripts can be used to directly execute redis commands on the server. General Data processing operations can be completed directly using the Lua language or the function library provided by the Lua interpreter without returning it to the client for processing.
2. all scripts are executed in the form of transactions. The script will not be interrupted by other work during execution, nor cause any competition conditions, you can use the Lua script to replace transactions and optimistic locks.
3. all scripts are reusable. That is to say, when repeated operations are performed, you only need to call the script cache stored in the server. You do not need to resend the entire script, to save network resources as much as possible.

Execute Lua script
Eval script numkeys key [Key...] Arg [arg...]
The script parameter is the Lua script to be executed.
Numkeys is the number of database keys to be processed by the script, and the subsequent key [Key…] The parameter specifies the database key to be processed by the script. The input key can be obtained by accessing the keys array in the script. For example, keys [1] retrieves the first Input key, keys [2] retrieves the second Input key, and so on.
Arg [Arg…] Parameters specify the parameters used by the script. In the script, you can access the argv array to obtain these parameters. Explicitly specifying the keys used in the script is used in combination with the redis cluster's key check. If this is not done, using the script in the cluster may cause errors.
In addition, you can easily reuse the same script by explicitly specifying the database keys and related parameters used by the script, rather than writing the database keys and parameters in the script.

Eval command example
Redis> eval "Return 'Hello world'" 0
"Hello World"
Redis> eval "return 1 + 1" 0
(Integer) 2
Redis> eval "Return {keys [1], keys [2], argv [1], argv [2]}" 2 "MSG" "Age" 123 "Hello World"
1) "MSG" # Keys [1]
2) "Age" # Keys [2]
3) "123" # argv [1]
4) "Hello World" # argv [2]

Execute the redis command in the Lua script
By calling the redis. Call () function or redis. pcall () function, we can directly execute the redis command in the Lua script.
Redis> eval "Return redis. Call ('ping')" 0 # Run the ping command in the Lua script
Pong
Redis> eval "Return redis. Call ('dbsize')" 0 # Run the dbsize command in the Lua script
(Integer) 4
# Execute the GET command in the Lua script to retrieve the MSG key value and concatenate the value with a string
Redis> set MSG "Hello World"
OK
Redis> eval "Return 'The message is: '.. redis. Call ('get', keys [1])'" 1 msg
"The message is: Hello World"

Difference between redis. Call () and redis. pcall ()
Redis. call () and redis. pcall () can be used to execute redis commands. The difference is that when the executed script fails, redis. call () will return the name of the error script and the error message of the eval command, while redis. pcall () Only returns the error message of the eval command.
Redis> eval "Return redis. Call ('notexistscommand')" 0
(Error) Err error running script (call to f_ddabd662fa0a8e105765181ee7606562c1e6f1ce ):
@ User_script: 1: @ user_script: 1: Unknown redis command called from Lua script
Redis> eval "Return redis. pcall ('notexistscommand')" 0
(Error) @ user_script: 1: Unknown redis command called from Lua script
In other words, when an error occurs in the executed script, redis. Call () can provide more detailed error information for troubleshooting.

Example: Use the Lua script to re-implement the zdecrby command
Create a zdecrby. Lua file that contains the following content:
Local old_score = redis. Call ('zscore ', keys [1], argv [2])
Local new_score = old_score-argv [1]
Return redis. Call ('zadd', keys [1], new_score, argv [2])
Run the following command to execute the script:
$ Redis-cli -- eval zdecrby. Lua salary, 300 Peter
(Integer) 0
This is equivalent to executing eval "Local... 1 salary 300 Peter has the same effect, but it is easier to save the script content to the file and then execute the script file. In addition, zdecrby implemented by this script is much simpler than zdecrby implemented by transactions and optimistic locks.

Use evalsha to reduce network resource loss
Any Lua script, once executed by the eval command, will be stored in the script cache of the server. You only need to use the evalsha command to specify the sha1 value of the script to be cached, you can execute the script again without sending the script:
Evalsha sha1 numkeys key [Key...] Arg [arg...]
Use the sha1 value to reuse the script that returns 'Hello world' information:
Redis> eval "Return 'Hello world'" 0
"Hello World"
Redis> evalsha 5332031c6b470dc5a0dd9b4bf2030dea6d65de91 0
"Hello World"
Use the sha1 value to reuse the previously implemented zdecrby command, so that the entire script does not need to be sent every time:
Redis> evalsha 918130cae39ff0759b8256910842f77300a91cb2 1 salary 500 Peter
(Integer) 0

Script management commands
Script exists sha1 [sha1...]
Check whether the script represented by the sha1 value has been added to the script cache. If yes, 1 is returned. If no, 0 is returned.
Script load script
Store the script in the script cache and wait for evalsha to be used in the future.
Script flush
Clear all scripts stored in the script cache.
Script kill
Kill the running timeout script. If the script has already executed the write operation, you also need to use the shutdown nosave command to force the server not to save the data, so that the wrong data is saved to the database.

Function library
Redis loads some common function libraries in the Lua environment. We can use these function libraries to process data directly in the script. They are standard libraries:
• Base Library: Contains Lua core functions, such as assert, tostring, error, and type.
• String Library: Contains functions used to process strings, such as find, format, Len, and reverse.
• Table Library: Contains functions used to process tables, such as Concat, insert, remove, and sort.
• Math Library: contains common mathematical computing functions, such as ABS, SQRT, and log.
• Debug Library: contains the functions required by the debugging program, such as sethook and gethook.
And external library
• Struct Library: converts the structure of the C language to the value of the Lua language.
• Cjson Library: converts a Lua value to a JSON object or a JSON object to a Lua value.
• Cmsgpack Library: encodes the Lua value into the messagepack format or decodes the Lua value from the messagepack format.
There is also an external function redis. sha1hex used to calculate the sha1 value.

Redis Lua script