Troubleshooting of a distributed lock automatic delay failure problem based on ETCD

Today, in testing the ETCD-based distributed lock process, the function of automatically extending the TTL after the test acquires the lock, exceeding the TTL length before releasing it, always returns a 404 error message when the TTL of the specified key is extended, and the target key does not exist when the TTL is updated on the target key.

The final troubleshooting is that the system time between the 3 nodes of the ETCD cluster is inconsistent because the TTL extension is performed in a separate listener thread after the key is created, and the TTL is updated after half the TTL, so the update TTL may occur before the target key is deleted due to the time ahead node in the cluster. The error that caused the target key to not be found when updating the TTL.

Troubleshooting after synchronizing all node system time in the cluster:

Ntpdate pool.ntp.org

Troubleshooting of a distributed lock automatic delay failure problem based on ETCD