A slow query problem that Redis must pay attention to

Today, the parsing service throws a timeout exception during querying of Redis's set data, and the method that produces the exception is:

Db. Setmembers (key);

The result of this API is to specify all the KV objects within the set, and the only way to solve this problem is to use another API:

Db. Setscan (key);

This API also returns all the KV objects within the set. Functionally, the 2 APIs are the same, but they return the object, the former is redisvalue[], and the latter is IEnumerable. But the results in today's actual scenario are different. From the return type can be seen, the former is a synchronous query API, and the latter is a lazy query, in today's actual scenario, set the amount of data is about 20+M, a single request synchronous query timeout is also normal. These are the reasons for the problem itself and the solution.

* * *

More in-depth, the execution mechanism of the two same-function APIs reflects the idea of the Stackexchange encapsulation API, considering that Redis is a single-threaded operation that will block other requests if it takes longer to perform operations in Redis. Stackexchange in a connectionmultiplexer package a TCP connection, before some earlier will also say that the cost of connectionmultiplexer is very high, generally in the form of a single case, This explains why Stackexchange directly throws a timeout on a slow query, rather than waiting for a timeout after the request, for the simple reason that a single TCP connection can handle only one request/response at a time. If a slow query is processed, subsequent request will be blocked, This is also evidenced by the stackexchange thrown in the timeout exception: The QS value indicates the number of request blocked by the TCP connection used by the current connectionmultiplexer. While Stackexchange API encapsulation is more "intelligent", or with the exception of today's api:setmembers to illustrate, in set data volume is small, normal return all kv no problem, but the amount of data a large, such as today 20+m situation, Stackexchange will not send this request directly and throw a timeout. This "intelligence" also poses a hidden danger to the project development test: Because of the small amount of data in the test, this problem will not be highlighted, but in the production environment there will be problems, and the amount of data to a certain extent before it appears. So the various slow queries of Redis, such as key *,keys and today's setmembers, are the hidden dangers that need to be handled with care. Keep in mind the Redis single-threaded features and try to control time-consuming slow queries to avoid reducing the overall performance of Redis. To reduce the Codereview effort for troubleshooting redis issues, you may also consider centralizing Redis-related operations in the project code structure.

A slow query problem that Redis must pay attention to