The implementation scheme of high concurrency core technology-idempotent

The implementation scheme of high concurrency core technology-idempotent

I. Background
we have a lot of operations in the actual system, no matter how many times, should produce the same effect or return the same result.
For example:

1. The front end repeatedly submits the selected data, should only produce a response to this data in the background.
2. We initiate a payment request, should only deduct the user account once the money, when encounters the network re-send or the system bug re-sends, should also only deduct once the money;
3. Send a message, should also be sent only once, the same text message to the user, the user will cry;
4. Create a business order, a business request can only be created one, creating multiple will be a big problem.

and so on many important cases, these logics need to be supported by Idempotent features.

Second, the concept of idempotent
Idempotent (idempotent, idempotence) is a mathematical and computer concept, common in abstract algebra.

In programming. The characteristic of a idempotent operation is that its arbitrary execution has the same effect as one execution. Idempotent functions, or idempotent methods, are functions that can be executed repeatedly with the same parameters and can obtain the same result. These functions do not affect the state of the system, and do not worry that repeated execution will cause changes to the system. For example, the "GetUserName () and Settrue ()" Functions are a idempotent function.

a more complex operation Idempotent Guarantee is achieved using a unique transaction number (serial number).

My understanding: idempotent is an operation, no matter how many times it is executed, the effect is the same as the return result .


III. Technical Programmes
1. Query Operations
Query once and query multiple times, in the case of the data unchanged, the query results are the same. Select is a natural power-like operation

2. Delete Operation
The delete operation is also idempotent, and deleting one and more times deletes the data. (Note that the result may not be the same, the deleted data does not exist, returns 0, the number of deleted data, multiple results returned)

3. Unique index to prevent new dirty data
For example: Alipay's funds account, Alipay also has a user account, each user can only have one fund account, how to prevent users to create funds account multiple, then to the Fund Account table in the user ID plus a unique index, so a user added success of a fund account record

Key points:
Unique index or unique composite index to prevent dirty data from being added to new data
(When the table has a unique index, concurrency when new error, then query once again, the data should already exist, return the results can be)

4. Token mechanism to prevent repeated page submissions
Business Requirements:
Page data can only be submitted once by clicking on the
cause of Occurrence:
Data is duplicated due to repeated clicks or network re-sends, or nginx re-delivery
Workaround:
Cluster environment: Using token plus Redis (redis single-threaded, processing requires queuing)
Single JVM Environment: use token plus Redis or token plus JVM memory
Processing Flow:
1. Before submitting the data to the service application Token,token to Redis or JVM memory, token valid time
2. Post check token after submission, delete token, generate new token return
Token Features:
To apply, once valid, can limit flow

Note: Redis will use the delete operation to determine token, delete successfully represents token verification pass, if using Select+delete to verify token, there is a concurrency problem, not recommended

5. Pessimistic lock
Lock acquisition when fetching data
SELECT * from table_xxx where id= ' xxx ' for update;
Note: The ID field must be a primary key or a unique index, or it's a lock table.
Pessimistic locks are used in conjunction with transactions, data locking time may be very long, according to the actual situation to choose

6. Optimistic lock
Optimistic lock is only in the moment of updating data lock table, other time does not lock the table, so relative to pessimistic lock, more efficient.

Optimistic locks can be implemented in a variety of ways through version or other state conditions:
1. Implementation by version number
Update table_xxx set name= #name #,version=version+1 where version= #version #
The following image (from the Internet):



2. Through conditional restrictions
Update table_xxx set avai_amount=avai_amount-#subAmount # where avai_amount-#subAmount # >= 0
Requirements: quality-#subQuality # >=, this scenario is suitable for no version number, only update is done data security check, suitable for inventory model, deduction share and rollback share, higher performance

Note: Optimistic lock update operation, preferably with a primary key or a unique index to update, this is a row lock, otherwise the update will lock the table, the above two SQL changed to the following two better
Update table_xxx set name= #name #,version=version+1 where id= #id # and version= #version #
Update table_xxx set avai_amount=avai_amount-#subAmount # where id= #id # and avai_amount-#subAmount # >= 0

7. Distributed lock
Or take the Insert data example, if the distribution is the system, it is difficult to build a global unique index, such as the uniqueness of the field is not determined, this time can introduce a distributed lock, through a third-party system (Redis or zookeeper), the business system to insert data or update data, to obtain a distributed lock, Then do the operation, then release the lock, this is actually the multi-threaded concurrency of the lock idea, the introduction of multiple systems, that is, distributed systems to solve the idea.

IMPORTANT: A long process process requires no concurrent execution, you can obtain a distributed lock based on a flag (user id+ suffix, etc.) before the process executes, and the lock will fail when the other process executes, that is, the process can only have one execution success at the same time, and after the execution completes, release the distributed lock ( Distributed locks to be provided by third-party systems)

8. Select + Insert
Concurrency is not high background system, or some task job, in order to support idempotent, support repeated execution, simple processing method is to first query the next key data, to determine whether it has been executed, in the business process, you can
Note: The core high concurrency process does not use this method

9. Power of State machine
In the design of documents related to the business, or the task-related business, will certainly involve the state machine (state change diagram), that is, the business document has a state, the state will change in different circumstances, the general existence of a finite state machine, at this time, if the state machine is already in the next state, At this time a change in the previous state, theoretically cannot be changed, so that the power of the finite state machine is guaranteed.

Note: Orders and other documents business, there is a long flow of state, it is necessary to understand the state machine, the business system design ability to improve a great help

10. How APIs that provide external interfaces ensure idempotent
If the payment interface provided by UnionPay: required to submit a payment request to the merchant, it comes with: source, SEQ serial number
Source+seq a unique index within the database to prevent multiple payments, (only one request can be processed when concurrent)

Key points:
External interface in order to support idempotent calls, the interface has two fields must be transmitted, one is source source, one is the source of the serial number SEQ, this two field in the provider system to do a federated unique index, so when the third party calls, first in the system inside the query, whether it has been processed, Returns the corresponding processing result, has not been processed, handles accordingly, returns the result. Note, in order to power-friendly, must first inquire, whether to deal with the pen business, do not query directly into the business system, will error, but the actual has been processed.


Summarize:
Idempotent should be a qualified programmer of a gene, in the design of the system, is the primary consideration, especially in such as Alipay, banks, internet finance companies are involved in the system of money, both efficient, accurate data, so can not appear more deductions, more dozen money problems, it will be difficult to handle, user experience is not good