Redis exists command takes a long time to troubleshoot

I. BACKGROUND

Redis Slow log analysis platform after the line, casually looked at, found Onestore used cache cluster, there are a large number of exists command slow query situation:

An average of 13msper exists command requires a maximum time of nearly 20ms. This is a very unscientific result. The exists command just executes a hash lookup operation, which should be the US level.

And the relevant students understand the business background as follows:

-Business is userfeed, storing the user's published news

-Use Zset to store all dynamic posted by a user, key is the user ID, and the corresponding Feedid in the collection. If users publish a lot of dynamic, Zset is also very large

-Redis cluster as Onestore cache, expiration time is 10 minutes

-exists is called before accessing the cache to see if it is hit and backfill the cache with Onestore if not hit

Because some users publish a lot of dynamic (2w+), so there are a lot of zadd slow queries.

Second, the investigation

1. Redis's policy to purge expired keys

-Passive mode: In the event loop, execute about 10 times per second, try to delete the expired key, there will be missed situation

-Random check of 20 keys from expire set

-Delete to expired key

-If more than 25% of the keys are out of date, repeat the first step (more than 25% of the expired key accounts are many)

-Active Mode:

-If the key is missing in the passive mode, check and clear when it is accessed again

2. View the Code

void Existscommand (Redisclient *c) {    expireifneeded (c->db,c->argv[1]);  Check if the key is out of date, delete if expired (    dbexists (c->db,c->argv[1)) {        addreply (c, Shared.cone)}, or    else {        addreply (c, Shared.czero);}    }

an active policy to purge expired keys is implemented in the EXISTS command handler, and the expireifneeded function is called to check if the key to be accessed expires and delete the key if it expires. The del command is a time-consuming operation when deleting a complex data type (list, hash, Zset, set) with many elements. Due to the existence of many elements of zset, like Zadd, the deletion of zset requires one to traverse all elements, the time complexity is large o (n). Because this delete operation is performed in the handler function of the exists command, the exists time is too long.

3. Redis Slow Log verification

through slow log can verify the above conclusions

-exists check the existence of ' user:94479529:feed ' first, the key has expired, triggering an active expiration mechanism, delete the key

-Retrieve the key data from Onestore and then backfill through Zadd

Third, the risk

The single instance of the cluster has a QPS of 8k+ and a 10w+ slow query every day. When there are a large number of slow queries on Redis, there is a case of an individual client timeout, which causes the request to fail.

Iv. Follow-up treatment

1. Increase the expiration time from 10 minutes to 20 minutes

2. For Redis cluster expansion (try to increase the effect of expiration time)

For Redis's time-consuming problem of removing large keys, the Redis author provides a solution that uses asynchronous threads to delete large keys to avoid blocking the main thread.

Redis exists command takes a long time to troubleshoot