Redis TTL is 0

Address: Http://get.jobdeer.com/7297.get A redis TTL of 0 troubleshooting

The thing is, this noon business suddenly to find me on the RTX, said a new Twemproxy cluster data query when there is a problem, Redis TTL returned to 0, let me help to see:

At that time, I felt the problem was very strange, according to previous experience, Redis TTL can not be 0 ah, see: Http://redis.io/commands/ttl

The Redis key, which returns the expiry time of key via the TTL command, is generally 3:

1. The current key does not set an expiration time, so it returns-1.

2. The current key has a set expiration time, and the key has expired, so it returns-2.

3. The current key has a set expiration time, and key has not expired, it will return the normal remaining time of key.

So, very puzzled why the key TTL of 0, then the first feeling problem will not appear in the twemproxy inside, so let the complex source development colleagues check twemproxy in the TTL command two times, at the same time log on to that Twemproxy , the TTL looks at the relevant key and confirms that the result is indeed 0, as shown in:

If this is the case, the first option is to insert the key test yourself:

The test process is as follows:

1. Setex a 10 1; Set a key A with an expiration time of 10s and a value of 1.

2. Use the TTL command to view the remaining expiration time of a, and the result is 6s.

3. Wait a moment, again TTL view, key expiration time is 0.

Indeed, it is not an individual phenomenon. At the same time, the source of colleague feedback, Twemproxy itself did not do any processing TTL command, so we through the internal Find_key tool, get the key is located on the hash ring on the real server (consistency hash algorithm), to where the Redis reconfirm:

It seems to be the problem of Redis, we began to suspect that the internal problems of Redis, and then tested on other versions, the results are correct, it appears that the version of the bug is very high, but it is not certain.

We did the same test on other instances of the same version, but we did not find that the TTL returned 0. It seems that only to check the source code.

So we looked at the source code for the Redis command for TTL, the code is as follows:

There is a case of TTL = 0 in the code, in theory a key that has an expiration time should return-2, and in this code, the first if statement (which should return-2) is not executed, which causes the second loop to be transferred into the On the current key expiration time must be less than the current timestamp (and not 1), so the TTL should be less than 0, and in the code, the author will ttl<0 the situation into ttl=0, the problem is why the first if not effective, Both the primary judgment function of the condition Lookupkeyread does not return null, and then the code of the function is viewed:

From this point of view, the function does not return NULL, and because the first if statement does not have a return null, it can be seen from the comments in the code that when Redis is a slave, it is possible not to return NULL.

As can be seen from the comments of the expireifneeded function, when the current Redis is slave, in order to ensure the consistency of the master-slave data, the current key is not deleted, triggering this sentence: if (server.masterhost! = NULL) return Now > when; The current time must be greater than the expiration time of the key store, so the function returns 1, so it goes back to Lookupkeyread, the function. The following function plays a decisive role:

The function will return NULL when the following conditions are met.

1. Current Link exists

2. The current link is not master

3. The currently linked command exists

4. The currently linked command is the same as true for Redis_cmd_readonly

The first three comparisons in the test process, it must be true, the problem in the fourth condition, here also leads to the Redis command flags, on the client, through the list of clients, you can view the current link to the flags:

As you can see, the flags for executing the TTL command are N, and in the code below you can see that flags=n is represented by flags=0, so in the above code, Flags & redis_cmd_readonly = 0 &2 (redis_cmd_ READONLY = 2,redis.h), so the IF statement is not entered, so it does not return null, resulting in the Ttlgenericcommand command returning the result of ttl=0. (As for how Redis uses these flags and the principles of the IF statement above, there is a need for more in-depth analysis, which is not explained here)

So, in this case, we know that if a redis is a slave, and the slave-read-only is set to OFF, and a key with a TTL is written, when key expires, the key is not deleted by Redis, and the TTL is always 0 after expiration.

With this judgment, we confirm on the Redis that the info command is slave, and the colleague who consulted the relevant deployment learned that there were multiple levels of replication and double write in the process of data migration, so the Redis slave was set to a writable state. At this point, the slaveof of slave is set to no one, both disconnected and the TTL of all expired keys is returned-2 again.

Therefore, the use of Redis children's shoes, note that in the case of service migration, such as the formation of multi-level replication chain, the relay on the expiration key read and write processing need to pay attention to the TTL problem, if you encounter a TTL return equals 0 when the first time to determine the problem.

Redis TTL is 0