Record Redis's HGETALL

There was no pitfall in the world, and more people fell into the trap. I have heard people say that Redis's HGETALL is a pitfall, but I don't believe it: no matter what it is, you must step on your own to give up. To be nice, it's hard to hear about the Yellow River, but it's hard to hear about it. The program running started to be very stable, so stable that I want to send all the HGETALL

There was no pitfall in the world, and more people fell into the trap. I have heard people say that Redis's HGETALL is a pitfall, but I don't believe it: no matter what it is, you must step on your own to give up. To be nice, it's hard to hear about the Yellow River, but it's hard to hear about it. The program running started to be very stable, so stable that I want to send all the HGETALL

There was no pitfall in the world, and more people fell into the trap.

I have heard people say that Redis's HGETALL is a pitfall, but I don't believe it: no matter what it is, you must step on your own to give up. To be nice, it's hard to hear about the Yellow River, but it's hard to hear about it.

At the beginning, the program runs very stably. I want to send all the characters that say HGETALL is a pitfall: Success! At this time, I forgot the dangerous existence like a frog in warm water. The time is like one day. suddenly one day needs to be changed, I have to extend the HASH data content from a dozen fields to more than one hundred fields, and use Pipelining to get the results of hundreds of HGETALL at a time. So I fell into the trap: the server went down.

Why? Redis is single-threaded! When it processes a request, other requests can only wait. Requests are usually processed quickly, but when we use HGETALL, We must traverse each field to obtain data. During this period, the CPU resources consumed are proportional to the number of fields. If PIPELINING is used, it is undoubtedly worse.

How can this problem be solved? Please allow me to give a formula for the problem:

PERFORMANCE = CPUs / OPERATIONs

That is to say, in this scenario, to improve performance, either increase the number of CPUs in the operation process or reduce the number of operations in the operation process. Specifically, I have come up with the following methods:

With Memcached

Redis storage method is not changed. Additionally, we use Memcached to implement a set of cache, which stores the hgetall hash originally needed in Redis. Of course, because Memcached stores strings, so when we store HASH, we actually store the serialized string after HASH, and then deserialize it when querying. Generally, the Memcached client driver can transparently implement the serialization and deserialization processes. The advantage of this solution is that Memcached supports multiple threads, so more CPUs can be involved in computing. At the same time, the corresponding operations will be reduced because every field does not need to be traversed. Of course, there are also many disadvantages, because a new cache layer is introduced, memory is wasted and the complexity is increased. In addition, even if we only need to obtain data from a few fields, we have to first query the complete data, then filter again, which is a waste of bandwidth. Of course, in this case, we can directly query Redis, But it undoubtedly increases some complexity.

By the way, Memcached supports Multiget and achieves Pipelining-like effects, but you must be very careful about the Memcached pitfalls, that is, the Mulitiget bottomless pit.

Serialization field Redundancy

When Redis stores HASH data, it stores another field named "all". Its content is serialized from the original HASH data. during actual query, only the redundant field HGET can be deserialized. The advantage of this scheme is that the original HGETALL operation is simplified to HGET by serializing field redundancy. That is to say, you no longer need to traverse every field in the HASH, so even if multiple CPUs cannot be involved in the operation, however, the number of operations is greatly reduced, so the performance improvement is still significant. Of course, the disadvantage is also obvious. Like all redundancy methods, this solution wastes a lot of memory.

Some people may ask, although the process of traversing fields is absent, the deserialization process is added, and the deserialization cost is usually high. Can this improve performance? The key to the problem is that the field traversal operation was completed on a single CPU, and then the deserialization operation, no matter what language, multi-process or multi-thread can be used to ensure that multiple CPUs are completed, so the overall performance is improved.

...

In addition, many people intuitively solve the problem by running multiple Redis instances. Indeed, this can increase the number of CPUs in the operation process and improve performance. However, HGETALL and PIPELINING often lead to a geometric explosion in the number of operations in the operation process, in contrast, the number of Redis instances we can increase is just a drop in the water, so this method in this example cannot completely solve the problem.

...

It is used to step on. You don't have to worry about falling in. Of course, the premise is that you can climb out on your own!

Original article address: record the HGETALL from Redis. Thank you for sharing it with me.