Build interesting applications with Go and redis, and build with goredis

Build interesting applications with Go and redis, and build with goredis

This article consists of four parts. The first part describes the functions, applications, and data structures of redis. The second part is to start using redis to build the lock. The third is to use redis to build an online user statistician. The fourth is to use redis to build an Automatic completion program.

First, introduce the features of redis. redis has a variety of different data structures available, including string, hash, list, ordered set, bitmap, Hyperloglog, geographic coordinate (GEO). It also has memory storage and Event Response Systems Based on Multiple Routes, ensuring the speed at which command requests are executed. Third, it has a variety of additional functions, such as transactions, lua scripts, key expiration mechanisms (Regular automatic deletion of keys), key elimination mechanisms, multiple persistence methods (AOF, RDB, RDB + AOF. In addition, it also has powerful support for multi-host functions. In the past few years, when redis was more than, people often said it was a toy database because it lacked support for multiple hosts, in recent versions, the multi-host function has been gradually added, for example, providing high availability for the master and slave nodes. In the latest version 4.0, it also provides a module extension system that uses redis as a memory platform. It provides some interfaces for users to use redis functions through interfaces, more new functions are provided on the basis of the original, allowing you to develop it.

Figure 1

Redis has multiple data structures. The first is the string, which is a bit similar to the C language string, but some functions are added on the basis of the C language. For example, the length record is used to obtain the length of a C string that is linear, redis adds a string length record function, allowing customers to get the length with constant complexity, without having to traverse the entire process and directly access it. There are also some binary security, which can store binary data directly in the database, not necessarily strings, such as some compressed files. Then it also has a memory pre-allocation system. When you want to frequently repair strings, it can reduce the number of times of Memory re-allocation through the memory pre-allocation policy and improve performance.

Figure 2

The second hash is similar to the map in the GO language. A key is mapped to another value, and each key is different. The complexity of getting a single key is constant, if necessary, you can obtain all key-value pairs at a time, but the performance is poor.

Figure 3

There is also a list. The list is a bit like a sequence of GO. The bottom layer is implemented by a double-ended linked list, so you can operate on its header or tail, which is a constant complexity, however, if you want to add or truncate the list or traverse it, it will be slow. Unlike the previous hash table, it allows repeated elements.

Figure 4

The next step is the set. The set stores multiple different elements in an unordered manner. The operations for the complexity of a single element are constants, and the speed is very fast. You can associate it with other sets, perform intersection and Union calculation.

Figure 5

Ordered Sets are rare. Each element in a set is composed of a score and members. Each member is ordered by the value. You can get the first two and the last two parts according to the Members. For example, you can get the element with the lowest score or the two elements with the highest score. The bottom layer of an ordered set is implemented using a skip table. Therefore, it is complicated to obtain a single element.

Figure 6

Bitmap is an array composed of a series of binary bits. The array has an index. You can operate binary bits based on the index. You can set the specified bit separately to obtain multiple digits in the specified range or count them.

Figure 7

Hyperloglog is complex. It has an algorithm that computes a series of binary bits and calculates the Count value. One feature is that even if you add hundreds of millions of elements to Hyperloglog, it counts hundreds of millions of elements and occupies 12 GB of memory. We can see how to use Hyperloglog to reduce memory usage.

Figure 8

In the last geographic location, you can store a longitude, latitude, and location name in redis, or perform a Range Calculation on it, or compute something within 100 or 5 kilometers in this location, friends who write LBS applications should be interested in this.

Figure 9

This is a typical example of a redis database. redis is a key-Value Pair database, and its keys are strings. Its values can be any of the data structures we have just introduced.

Figure 10

We can use commands to operate these key-value pairs, just as we operate relational databases in SQL. First, you need to have a command to execute what you want, and the second one has a key, the key is to have an object you want to operate on, and then you need to have any multiple parameters, because it is very simple, sometimes it is often done with 1 or 2 parameters. Some complicated commands also have options and values.

Figure 11

Here are some command examples, such as ping. If all servers are normal, A ping will be returned for you, which is sorted in the form of a key, the third is to set Hash hash and multiple key-value pairs. The last RPUSH can add all the elements to the list.

GO back to our theme GO today. To use redis in GO, we need to use the client. The client officially recommends two radix and redigo. Today we will use radix and our old friend GOget. Here is the sample code for connecting to the client. Use the dial method to connect to the database, run the ping command using the cmd method, and obtain the reply. Finally, use the str method to convert the reply to a string, then print it out.

Figure 12

Come to our first application instance-lock. A lock is a synchronization mechanism that ensures that a resource can only be used by one process at any time. If other processes want to use this resource, they must wait, until the process using the resource gives up the right to use the resource. Generally, a lock has two operations: Get and release. The get operation is to obtain the right to use the resource. At any time, the security of the resource can only be obtained by one process, when a lock is obtained, other processes that attempt to obtain the lock will fail. There is also a release operation. After the lock process is released, other people can use the resource again.

The first method is to use the string structure to implement the lock. The specific method is to use a string as the lock. If the string key has a value, it indicates that the lock has been obtained, if the key has no value, it is not obtained.

The following are the required commands:

• GET key: GET the value of the character key. If the key does not have a value, a null value (nil) is returned)

• SET key value: SET the value of the character key to value. If the key already has a value, the new value will overwrite the old value by default.

• DEL key: delete a given key.

First, we use the GET method to obtain the key value, convert the value to a string, and then use the if method to check whether there is a value. if there is no value, an empty string is returned, if there is no value, call the set method to set it, that is, lock it. The code shown here is to save time, so we can process the error response.

Figure 13

Although this method can be successful, it has a competitive condition that there is an intermediate time period after the get command is executed and before the set method is executed, other clients may be preemptible to key settings. In this case, there will be a race condition. If there are two clients, they will execute the acquire method together and they will call get at the same time, all obtained the consensus that the key does not have a value. Client 2 executes the set Method to lock the execution because the execution speed is faster, and then the lock is successfully obtained. At this time, client 1 is late, because the get method is executed at the same time, we all think that the key is not locked, so client 1 will continue to execute this set method to lock the lock, at this time, two clients will get the lock at the same time, and an error will occur.

To solve this problem, we need to use the Transaction Features of redis to ensure the security of the lock operation. Here is a demo of redis non-transaction commands. Generally, many databases share the same structure. If you set a General Command, the transaction is a special command of the command, A transaction contains multiple commands to ensure security.

Now let's take a look at what commands are needed for a secure lock? The first one is WATCH, which monitors the given key. If the monitored key has been first modified by another client before the transaction is executed, the command submitted by the client that executes the command will be rejected. The second is the MULTI command, which starts a transaction. After executing this command, all the operation commands sent by the client will be entered into the transaction queue and waited. The last EXEC is to attempt to execute the transaction. If the transaction succeeds, a queue consisting of multiple command replies will be returned to the client. If the transaction fails, nil will be returned.

Figure 14

This is the implementation code. First, use the WATCH command to monitor a key and then call the get method. If the key has no value, call MULTI to start a transaction and put EXEC in the queue, this is the most critical step. if the transaction is successfully executed, the response will not be blank. If the returned result is null, it indicates that the lock_key has been first modified by other clients, then we can determine whether the lock is successful based on whether the lock is successfully executed.

However, the use of transactions will also bring about a price, which will complicate the code. Our lock program itself has less than five lines of code. Now it has been added to 10 lines, and the amount of code has been doubled, although the security is ensured, the code is complicated. Considering this situation, redis provides a new feature-the SET command with the NX option. When we use the SET command with the NX option, it is SET only when the key does not exist. If the key already has a value, it will discard the setting code and return nil to indicate that the setting failed. The role of the NX option is to put the code that will cause competition in the previous paragraph into the server for execution, thus ensuring the security of the execution operation.

Figure 15

This is our NX option. It is very simple. It is simpler than the lock of the original version. It is a SET command, but there is a NX option, when the lock_key has no value, you can set it. By checking whether the reply is empty, you will know that the lock is successful.

Figure 16

Take a look at the second application example-online user statistician, for example, you can count the number of users on the website and the number of people watching the live video. The first way to implement this function is to use a set. When a user goes online, we add the user name to the online user set.

Figure 17

The required command, SADD set element [element...], can add the given element to the set. SCARD set can obtain the base number of the set, that is, the number of elements contained in the set, that is, the number of users currently. SISMEMBER set element can check whether a given element exists in the set. For example, you can use this command to check whether a user is online.

Figure 18

The implementation code is also very simple. When a user is online, we accept the user name and call the SADD command. to count how many users are online, we can obtain the base number of elements in the set, finally, if we want to check whether a user is online, 1 will be returned when the given element exists. We can check whether the user is online when the number of returned values is 1.

Using set statistics online users has a very serious problem. The volume of the set will increase with the increase of elements. Assuming that the average name of each user is 10 bytes, therefore, websites with 1 million users need to use 10 MB of memory for storage every day. websites with 10 million users need to use 100 MB of memory for storage every day. If you store this information for one year, websites with 1 million users need to use 10 million GB of memory each year. websites with million users need to pay GB of memory each year. Calculating how much memory is consumed for such a small function is not worthwhile, and does not include any additional overhead.

The second method is to use bitmap to create an ID for each user. When a user goes online, his ID is used as the index. Assume that there is a user peter, we map an ID 10086 to the bitmap, and set the index value of 10086 to 1 Based on the ID. the user with the value of 1 is online, if the value is 0, it is not online.

Three commands are also required here:

• SETBIT bitmap index value: sets the binary bit on the specified bitmap index to the specified value.
  

• GETBIT bitmap index: obtains the binary bit of the specified bitmap index.

• BITCOUNT bitmap: count the number of binary bits whose value is 1 in the bitmap.

The implementation code is as follows. setbit accepts the user ID and sets the binary bit to 1. The number of binary bits whose statistical value is 1. to check whether the user is online, first check whether the binary bits of the bit watermark are online according to the user ID, 0 is not online.

Figure 19

Compared with the preceding set, although the size of the bitmap still increases with the increase in the number of users, the size of the memory required for each user is changed from the average 10 bytes to one bit, therefore, it will save a lot of memory and reduce the usage of dozens of GB to several hundred MB.

To continue optimization, We can get method 3: Use Hyperloglog. When a user goes online, Hyperloglog is used to count the user. Assume that there is a user jack who uses the Hyperloglog algorithm to count and then reflects the count to Hyperloglog. If this element has not been counted by Hyperloglog before, when you add a new Hyperloglog, you will add 1 to your count. If jack already exists, it will not add 1 to its Count value.

Hyperloglog requires two methods. The first method is PFADD hll element [element...] to count the given elements. The second PFCOUNT hll is used to obtain the approximate base of Hyperloglog, that is, the base estimation value. Hyperloglog has one defect because Hyperloglog is a probability algorithm and can only provide one estimated value. For example, if you have 1000 users for counting, you may only return 970 or 980. It cannot provide an accurate counting value, but can only give an approximate value, the benefit is that no matter how many users we count, a single Hyperloglog only occupies 12 kb. The following is the implementation code.

Figure 20

The content consumption pairs of the three implementations are as follows:

Figure 21

The last example is auto-completion. We have many desktop applications, such as web browsers, search engines, and twitter. When you enter something, for example, if you enter go in your browser, it will automatically help you complete some go-related commands. Let's analyze the principle of auto-completion. When we input it, we will return a result of auto-completion. This result also carries a weight value, which is higher than the previous value. To implement this auto-completion program, we need to build a weight table. For redis, the most suitable for storing such weights is an ordered set.

Figure 22

How can we create a weight table? There are many ways to build a weight table. For example, you can use many complicated algorithms to calculate the weights of each candidate structure. There are also some simple methods, for example, the simplest thing is that you enter gmail based on the user's input, and then you count the user. If you enter google for another count, then, a weight table is constructed based on the customer input count. Suppose we have a weight table with a gopher weight of 277. If we input a gopher three times in a row, the gopher weight is increased to 280. So we built a weight table, but we cannot complete it. If we want to complete it, you will think of GOPHER when you input G, and gopher when you input go. We need to give such an arrangement, create a weight table for each sort. For example, add a gopher entry to the weight table and add a weight item for each sort. Whenever a user inputs a gopher, we find the gopher in the weight table for each weight table corresponding to the arrangement, and then add the gopher.

Figure 23

During the completion process, when we input G, we will search for the corresponding weight table based on G. We can see that gopher ranks third, and its score is lower than that of google and gmail, this table contains all the candidate results starting with G. When we enter go, we start to associate it. It will continue to look for Members starting with go. We can see that gopher is constantly rising and appears on the user base. When we enter GOP, we will find our gopher Based on GOP.

Figure 24

Two commands are required to implement auto-completion. The first ZINCRBY zset increment member is to perform auto-increment operations on the score of a given member; the second ZREVRANGE zset start end [WITHSCORES] obtains the members within the specified index range from the ordered set in the order of values from the largest to the smallest. Because our weights are arranged from large to small, we will first get the value with the highest weight to be displayed. The implementation code is as follows.

Figure 25

Let's sum up: first, GO and redis are both very simple and powerful tools. They can be combined to easily solve many features that are often difficult to implement or require a lot of code to implement, such as automatic completion, if you do not use GO and redis, the number of PPT pages today may increase by three times. The second point is to ensure the security and correctness of the program when building the program. Although these two points often make the program complex, sometimes there are solutions that both fish and bear's paw have their own advantages. For example, the online statistics mentioned above can be implemented in many ways, but the optimal method can be found only when you are familiar with it. The last point is that the efficiency and functions of different methods are usually different. We need to choose based on our own situation and do not blindly trust the so-called optimal solution.