What are the features of Redis that solve the problem?

First look at what Redis is. The official profile explains:

Redis is a BSD open source project, a storage system that puts structured data in memory, and you can use it as a database, cache, and message middleware. Data types such as strings,lists,hashes,sets,sorted sets,bitmaps,hyperloglogs and geospatial indexes are also supported. It also has built-in replication, LUA scripting, LRU, transaction functions, high availability through Redis Sentinel, and automatic sharding via Redis cluster. As well as transactions, publish/subscribe, automatic failover and so on.
In summary, Redis provides a wealth of features, the first time you may feel dazzled, what are these features for? What's the problem solved? Under what circumstances will the corresponding function be used? Then the following is a rough explanation of the evolution from the zero-step to the next.

1 Start from scratch

The initial demand was very simple, and we had a API:HTTP://API.XXX.COM/HOT-NEWS,API consumer complaining about a list of hot news reports that each request would take about 2 seconds to return the result.

Then we started with how to improve the performance of API consumer perception, and soon the simplest and most brutal first scenario came out: for the API Response plus HTTP-based cache control cache-control:max-age=600, which allows consumers to cache this response for 10 minutes.

If the API consumer effectively leverages the cache control information in the response, it can effectively improve its perceived performance (within 10 minutes). But there are 2 drawbacks: the first one is that the API consumer may get the old data within 10 minutes of the cache, and the second is that if the API's client ignores the cache, the direct Access API still takes 2 seconds, and the symptoms don't.

2 native memory-based caching

In order to solve the call API still need 2 seconds, after troubleshooting, mainly because of the use of SQL to get hot news in the process of consuming nearly 2 seconds, so we think of a simple rough solution, That is, the results of the SQL query are cached directly in the current API server's memory (set cache validity time is 1 minutes). In the next 1 minutes, the request reads the cache directly, no longer takes 2 seconds to execute the SQL.

If the API receives 100 requests per second, then a minute is 6,000, which means that only the first 2 seconds of congestion will take 2 seconds, and all subsequent requests in 58 seconds can be made even if the response is not required to wait 2 seconds.

The other API's small partners found this to be a good idea, and soon we found that the API server's memory was full ...

3 Redis on the service side

When the memory of the API server was full, we found that we had to think of another solution. The most straightforward idea is that we throw all these caches onto a dedicated server, and put it in a large memory configuration. And then we got a look at Redis ... As for how to configure the deployment Redis here is not explained, the Redis official detailed introduction. We then used a separate server as the Redis server, and the memory pressure of the API server was resolved.

3.1 Persistence (persistence)

Single Redis Server One months there is always a bad mood, bad mood on the strike, resulting in all the cache is lost (redis data is stored in memory). Although the Redis server can be back online, but due to the loss of memory data, resulting in a cache avalanche, API server and database pressure is coming up in a sudden. So this time, the Redis persistence feature comes in handy to mitigate the impact of the cache avalanche. Redis's persistence means that Redis writes data from memory to the hard disk and loads the data when the Redis restarts, minimizing the impact of cache loss.

3.2 Sentinel (Sentinel) and Replication (Replication)

Redis Server A strike without warning is a nuisance. So, what's the run? Answer: Backup One, you hang it on. So how to know a Redis server hangs, how to switch, how to ensure that the backup machine is the original server full backup? Sentinel and replication are required to play. Sentinel can manage multiple Redis servers, which provide monitoring, alerting, and automatic failover capabilities; Replication is the server that enables a Redis server to be equipped with multiple backups. Redis also uses these two features to ensure that Redis is highly available. In addition, Sentinel functionality is a leveraging of Redis's publishing and subscription capabilities.

3.3 Clusters (Cluster)

There is always an upper limit for a single server resource, CPU resources and IO resources we can transfer a portion of the CPU and IO pressure to the slave server through master-slave replication and read/write separation. However, the memory resources, master-slave mode is only the same data backup, and can not expand the memory horizontally, the memory of a single machine can only be increased processing, but there is always a limit. So we need a solution that allows us to scale horizontally. The ultimate goal is to take each server only as a part of it, so that all of these servers form a whole, for the outside consumer, This set of distributed servers is like a centralized server (previously explained in the rest blog about distributed network-based differences: architecture based on Web applications).

Before the Redis official distributed solution came out, there are twemproxy and Codis two scenarios, these two scenarios are generally dependent on proxy for distributed, that is, Redis itself does not care about the distribution of things, But by Twemproxy and Codis. The cluster scheme, which is officially given by Redis, is to make the distributed part of each Redis server, so that it no longer needs other components to complete the distributed requirements independently. We don't care about the advantages of these programs here, so let's take a look at what the distribution here is all about. What is the problem with this part of the logic that Twemproxy and Codis independently handle and the logic that cluster integrates into the Redis service?

As we said earlier, a distributed service looks like a centralized service to the outside world. Then there is a problem to be solved: to increase or decrease the number of servers in a distributed service, which should not be felt for clients consuming the service, then it means that the client cannot penetrate the distributed service and bind itself to a server of a certain station, because once that happens, You will no longer be able to add servers and fail to replace them.

There are two ways to solve this problem:

The first way is the most direct, that is, I add a middle layer to isolate this specific dependency, that is, twemproxy, so that all clients can only through it to consume the REDSI service, through it to isolate this dependency (but you will find that Twermproxy will become a single point), In this case, each Redis server is independent, and they do not know each other's existence;

The second way is to let the Redis server know each other's existence, through the redirection mechanism to guide the client to do what they want to do, such as the client link to a Redis server, said I want to do this operation, Redis server found itself unable to complete this operation, Then you will be able to complete this operation of the server information to the client, let the client to request another server, this time you can find each Redis server needs to maintain a complete distributed server information, a copy of the data, How else would it know to ask the client to find the other server to perform the client's desired operation?

This big paragraph above explains so much, I do not know whether it is the first path or the second path, there is a common thing exists, that is distributed services all the servers and the services they can provide information. This information is to exist anyway, the difference is that the first way is to manage this part of the information separately, use this information to coordinate the back end of the multiple independent Redis server, the second way is to let each Redis server hold this information, each other know each other's existence, To achieve the same purpose as the first path, the advantage is that no additional components are needed to handle this part of the story.
The specific implementation details of the Redis cluster are the concept of a hash slot, which is a pre-allocated 16,384 slots: The client gets the corresponding slot by CRC16 (key) 16384 operation on the key. On the Redis server is a subset of the slots that are responsible for each of the servers, and when a new server is added or removed, the slots and their corresponding data are migrated, and each server holds the full slot and its corresponding server information, which allows the server side to redirect requests to the client.

4 Redis for clients

The third section above focuses on the evolution steps of the Redis server, explaining how Redis can be transformed from a single stand-alone service into a highly available, decentralized, and distributed storage System. This section is concerned with Redis services that clients can consume.

4.1 Data types

Redis supports a wide range of data types, from the most basic string to the most complex and commonly used data structures:

String: The most basic data type, a binary security string, and a maximum of 512M.
List: Lists of strings in order of the order in which they are added.
Set: Unordered collection of strings, no duplicate elements exist.
Sorted set: A sorted collection of strings.
A collection of hash:key-value pairs.
Bitmap: A more granular operation, in bits.
Hyperloglog: Probabilistic-based data structures.
These numerous data types are primarily designed to support the needs of a variety of scenarios, although each type has a different time complexity. In fact, these complex data structures are equivalent to the specific implementation of remote data access, which I have described in the Web-based architecture style of the "Reading Rest" blog series, that is, by executing a standard set of operations commands on the server, Get the desired narrowing result set between the server, which simplifies the use of the client and improves network performance. For example, if you do not have a list of this data structure, you can only save the list into a string, the client to get the full list, the operation and then complete the submission to Redis, will produce a lot of waste.

4.2 Business

Each of these data types has a separate command to operate, and in many cases we need to execute more than one command at a time and need to succeed or fail at the same time. Redis's support for transactions also stems from the need to support the ability to execute multiple commands sequentially and to ensure their atomicity.

4.3 Lua script

On the basis of the transaction, if we need to perform more complex operations (including some logical judgments) on the server side, then Lua can come in handy (for example, to get a cache and extend its expiration time). Redis guarantees the atomicity of LUA scripts, which in certain scenarios can be substituted for the transaction-related commands provided by Redis. Corresponds to a specific implementation of remote Evluation = REV, which is described in the architecture style of Web-based applications.

4.4 Piping

Because Redis's client and server connections are based on TCP, the default is to execute only one command per connection. Pipelines allow multiple commands to be processed with a single connection, saving some of the overhead of TCP connections. The difference between a pipeline and a transaction is that the pipeline is meant to save the overhead of communication, but it does not guarantee atomicity.

4.5 Distributed Locks

The official recommendation is to use the Redlock algorithm, which uses a string type, to give a specific key when locked, and then to set a random value, and then use LUA script to perform a fetch comparison before removing the key. The specific commands are as follows:

SET resource_name my_random_value NX PX 30000

If Redis.call ("Get", keys[1]) = = Argv[1] Then
Return Redis.call ("Del", Keys[1])
Else
return 0
End
Summarize

This article focuses on the abstraction to explain the functions of Redis and the purpose of its existence, without concern for its specific details. In order to focus on the problems they solve, the concept of abstraction can enable us to choose a more appropriate solution in a particular scenario, rather than limiting it to its technical details.

The above are some of the author's personal understanding, if inappropriate, please correct me.

What are the features of Redis that solve the problem?