Non-idempotent problem of API interface and simple distributed lock using Redis

In our platform, some interfaces are opened to other systems or third-party application calls.

Idempotent problem is mainly in the creation of objects and these objects can not be duplicated, such as creating users, this time, should say idempotent is not a problem, non-idempotent is the problem.

Create objects, generally using the HTTP POST method, in restful design, the post method is non-idempotent, but we use the Post method is not necessarily non-idempotent. For example, internal calls to ourselves can be created repeatedly (not idempotent), but open to the outside, cannot be created (idempotent).

When creating objects, what should be done to avoid problems arising from non-idempotent.

The request should be marked with an identity that is unique and credible. After receiving the request, query whether the identity already exists, create an object if it does not exist, and notify if it exists.

For example, using a mobile phone number to create a user, this mobile phone number can be used as the identity of creating users, if you want to be credible, send a verification code verification. When created, the background must first verify that a mobile phone has been registered, which prevents duplicate registration.

In a clustered or distributed environment, there is still a problem, although there is a unique identity in the request. For example, two requests in a row, quickly and quickly, all using the same identity, then the process is not good to create two objects. Because the first request does not save the object, the second request discovery has not yet been created and an object is created. Well, we're having this unprofessional problem.

Two requests may be performed on different application servers, the two objects created are stored on different databases, no locks can be used, no distributed transactions, and operation timing is not guaranteed.

This problem can be resolved by using distributed coordination services such as ZOOKEEPER/ETCD. We do not have such a service at the moment, so we can only use other means and, of course, find a suitable third party to ensure it. For distributed concurrency lock processing, the core is the need for a third party, the Arbiter, to meet:

1. The only one. Only one primary node in the cluster provides quorum at the same time.

2. Reliable. Can not say hanging on the hanging, the general use of clusters, at least to master prepared. When the primary node is hung, the data is not lost when the client connects to the other nodes.

3. Fast enough. or called high performance.

Then we found Redis, which is readily available, simple, and easy to solve in this case, because Redis services are common and are already deployed in your project. Using Redis's Incr method can help us solve this recurring creation problem. Using the same redis (master-slave), its atomicity guarantees that we don't need to worry about concurrency. When created, the unique identifier is used as a key and incr, and the return value, if 1, indicates that the object has not been created, and if it is greater than 1, it is already created. Meanwhile, the key cache time guarantees that the object can be saved to the database.

In the case of master preparation, it is not reliable to handle distributed locks unless Redis writes to the master repository and returns results (strong consistency) while the data is written to the repository. But Redis's primary and standby synchronization seems to be asynchronous, weak consistency synchronization, the possible problem is that the first client has just written to the Redis master node is not synchronized to the standby node is hung, the second client had to access the Redis, found no data (no lock), and repeated acquisition of locks, In this way there are two clients that get to the lock.

Considering the non-idempotent problem in the Open interface, it is necessary to consider the distributed environment and prevent duplicate object creation, which is the problem and the solution.