Transaction Processing of Distributed Systems (recommended)

When we use a server on the production line to provide data services, I will encounter the following two problems:

1) the performance of a server is insufficient to provide sufficient capabilities to serve all network requests.

2) We are always afraid that our server will be shut down, resulting in service unavailability or data loss.

So we had to expand our servers, add more machines to share performance issues, and solve spof problems. We usually use two methods to expand our data services:

1)Data Partition: Data is divided into different servers (such as uid % 16 and consistent hash ).

2)Data Image: Make all servers have the same data and provide equivalent services.

In the first case, we cannot solve the data loss problem. When a single server fails, some data will be lost. So,The high availability of data services can only be achieved through the second method-redundant data storage(Generally, the industry thinks that the number of secure backups should be three, such as hadoop and Dynamo). However, adding more machines will complicate our data services, especially cross-server transaction processing, that is, cross-server data consistency.. This is a very difficult problem. Let's use the most typical use case: "account a transfers money to account B" to explain that anyone familiar with RDBMS transactions knows that six operations are required from account A to account B:

1. Read the balance from account.
2. Perform the subtraction operation on account.
3. Write the result back to account.
4. Read the balance from account B.
5. Add Account B.
6. Write the result back to account B.

For data consistency, both the six tasks are successfully completed or fail, and other accesses to accounts a and B must be locked, the so-called lock is to exclude other read/write operations, otherwise there will be dirty data problems, this is the transaction. Then, after we add more machines, this will become complicated:

 

1)In the Data Partition solution: What if the data of account A and Account B is not on the same server? We need a cross-machine transaction processing. That is to say, if a successfully deducts money but B fails to add money, We have to roll back the operation of. In this case, it becomes more complex.

2)In the data image solution: The remittance between account A and Account B can be completed on one machine, but do not forget that we have multiple machines with copies of account A and Account B. If there are two concurrent operations (to be remitted to B and C) for remittance to account a, what should I do if these two operations happen on different two servers? That is to say, in a data image, how can I ensure the consistency of write operations on the same data on different servers to ensure that data does not conflict?

At the same time, we also need to consider the Performance Factor. If we do not consider the performance, it is not difficult to guarantee the transaction, and the system will be slower. In addition to performance, we also need to consider availability. That is to say, if one machine is lost and data is not lost, services can be provided by other machines. Therefore, we need to consider the following situations:

1)Disaster Tolerance: No data loss, node failover

2)Data Consistency: Transaction Processing

3)Performance: throughput, response time

As mentioned above, to avoid data loss, you can only use data redundancy. Even if it is a data partition, data redundancy must be performed in each partition. This is the data copy: when the data of a node is lost, it can be read from the copy. The data copy is the only way to solve the data loss exception in the distributed system. Therefore, in this article, we will only discuss data consistency and performance when data redundancy occurs. Simply put:

1) to make data highly available, you have to write multiple data copies.

2) Data Consistency may occur when multiple copies are written.

3) Data Consistency may cause performance problems.

This is software development.

Consistency Model

Speaking of data consistency, there are three types (if subdivided, there are also many consistent models, such as sequence consistency, FIFO consistency, session consistency, and single-read consistency, single-write consistency, but for the sake of simplicity and readability in this article, I will only talk about the following three types ):

1)Weak Consistency: After you write a new value, the read operation may or may not be able to read the data copy. For example, some cache systems, the data of other players in online games have nothing to do with you, such as VoIP systems, or Baidu search engines ).

2)Eventually eventual consistency: When you write a new value, you may not be able to read it, but you will be able to read it after a certain time window. Such as DNS, email, Amazon S3, and Google search engine.

3)Strong strong consistency: Once new data is written, the new value can be read at any time in any copy. For example, file systems, RDBMS, and azure tables are strongly consistent.

From the three consistent models, we can see that weak and eventually are asynchronous redundancy in general, and strong is synchronous redundancy in general. Asynchronous Communication usually means better performance, but it also means more complex state control. Synchronization means simplicity, but also performance degradation. Well, let's look at the technologies step by step:

Master-slave

The first is the master-slave structure. For this addition, slave is generally a master backup. In such a system, it is generally designed as follows:

1) read/write requests are all handled by the master.

2) After the write request is written to the master node, the master node synchronizes the request to the slave node.

To synchronize data from the master node to the slave node, you can use asynchronous mode or synchronous mode. You can use the master node to push data or use slave to pull. Generally, it is the periodic pull of slave, so it is the final consistency. The problem with this design is that if the master crashes in the pull cycle, the data in the time slice will be lost. If you do not want to discard the data, slave can only be read-only for Master recovery.

Of course, if you can tolerate data loss, you can immediately ask slave to replace the master (for nodes only responsible for computing, there is no question about data consistency and data loss, the master-slave method can solve the single point of failure) Of course, The Master Slave can also be highly consistent, for example, when we write the master, the master is responsible for writing the first, after the write is successful, write the slave again. If both are successful, the return is successful. The entire process is synchronized. If the write slave fails, the two methods are available, one is to mark the slave unavailability and report an error and continue the Service (when the slave recovers and synchronizes the master data, there can be multiple slave, so that one less, there is also a slave, as mentioned above), the other is to roll back itself and return a write failure. (Note: Generally, do not write slave first, because if you fail to write the master, you must roll back the slave. In this case, if the slave fails to be rolled back, you have to manually correct the data) as you can see, if the master-slave requires strong consistency, how complicated it is.

Master-Master

Master-master, also called multi-master, refers to the existence of two or more masters in a system. Each master provides the read-write service. This model is an enhanced version of master-slave. Data Synchronization is typically done asynchronously between masters, so it is eventually consistent. The advantage of the master-master is that a master is down and other masters can perform read and write services normally. Like the master-slave, when data is not copied to another master, data will be lost. Many databases support the master-master replication mechanism.

In addition, if multiple masters modify the same data, this model has a nightmare-conflict merging between data, which is not easy. Looking at the design of Dynamo's vector clock (the version number and modifier of the record data), we can see that this is not that simple, and Dynamo's data conflict is handed over to the user. Just like our SVN source code, conflicts with the same line of code can only be handled by developers themselves. (The vector clock of Dynamo will be discussed later in this article)

Two/three phase commit

This Protocol is abbreviated as 2 PC, and Chinese is called two-phase commit. In a distributed system, although each node knows that its operation is successful or failed, it cannot know that the operation of other nodes is successful or failed. When a transaction spans multiple nodes,CoordinatorTo control all nodes in a unified manner (calledParticipants), And finally indicates whether these nodes need to submit the operation results (such as writing the updated data to the disk ). The two-phase commit algorithm is as follows:

Stage 1:

1. The Coordinator will ask all the participant nodes if they can perform the submit operation.
2. Preparations for each participant to start transaction execution, such as locking resources, reserving resources, and writing Undo/Redo logs ......
3. The participant responds to the Coordinator. If the transaction preparation is successful, the response "yes" is returned; otherwise, the response "no submission" is returned ".

Stage 2:

• If all participants respond to "yes", the Coordinator sends the "formally submit" command to all participants. The participant completes the formal submission, releases all resources, and then responds to "complete". The Coordinator collects the "complete" response of each node and ends the global transaction.

• If one participant responds to the "reject submission" request, the Coordinator sends a "rollback operation" to all participants, releases all resources, and then responds to the "rollback completed" request ", the Coordinator collects the "rollback" response of each node and then cancels this global transaction.

We can see that 2 PC stands for vote in the first stage, and an algorithm used to make decisions in the second stage. We can also see that 2 PC is a strongly consistent algorithm. We have discussed master-slave's strong consistency policy earlier, which is a bit similar to 2 PC, but 2 PC is more conservative-try to submit it first. 2 pcs use a lot of resources. In some system designs, a series of calls are connected, such as a-> B-> C-> D, each step allocates resources or modifies some data. For example, we have a series of processes in the background for B2C online shopping order operations. If we do this step by step, we will have such a question. If one step cannot be done, we need to perform a reverse operation to recycle all the previously allocated resources, therefore, operations are complicated. Currently, many processing flows (workflow) use the try-> confirm algorithm to ensure that the entire process can be completed successfully. For example, when western churches get married, they all have such bridges:

1) The priest asked the groom and the bride respectively: Do you want ...... No matter whether you are active or not ...... (Inquiry stage)

2) When both the groom and the bride answer the question (Lock resources for a lifetime), the priest will say, "I declare you ...... (Transaction commit)

This is a classic two-phase commit transaction processing. In addition, we can also see some of these problems. A) one of them is synchronous blocking operations, which will inevitably greatly affect performance. B) another major problem is timeout. For example,

1) if a participant does not receive an inquiry request during the first stage, or the participant's response does not reach the facilitator. Then, the Coordinator must handle the timeout. Once the timeout occurs, it can be regarded as a failure or retry.

2) if, in the second stage, after the formal submission is sent, if some participants do not receive the message, or the confirmation information after the participant submits/rolls back is not returned, once the participant's response times out, or retry, you can either mark the participant as the problematic node and remove the entire cluster. This ensures that the Service nodes are data consistent.

3) The worst case is that, in the second stage, if the participant fails to receive the commit/fallback command from the Coordinator, the participant will be in the "unknown state" stage, and the participant has no idea what to do, for example: if all the participants have completed the first-stage reply (all may be yes, all may be no, and some may be yes, some may be no), if the Coordinator fails at this time. Then all the nodes have no idea what to do (no other participants can ask ). To ensure consistency, you must wait for the Coordinator to either resend the first-stage yes/no command.

The biggest problem for submitting two paragraphs is 3rd,If a participant does not receive a decision in the second stage after the first stage is complete, the data node enters the "overwhelmed" state, which blocks the entire transaction.. That is to say, coordinator is very important for the completion of the transaction, and the availability of coordinator is critical. For some reason, we introduce the Three-segment commit. The description of the Three-segment commit on Wikipedia is as follows. He splits the first segment break submitted by the two-segment into two segments: query, and then lock the resource. Finally, submit the job. The three paragraphs are submitted as follows:

The core idea of submitting the three paragraphs is:The resource is not locked at the time of inquiry. The resource is locked only when everyone agrees..

Theoretically, if all the nodes in the first stage return success, there is a reason to believe that the probability of successful submission is high. In this way, the probability of unknown status of the cohorts participant can be reduced. That is to say, once the participant receives the precommit, it means that he knows that everyone actually agrees to the modification. This is important. Next let's take a look at the status migration diagram of 3 PC :(Note the dotted lines in the figure. The F and t values are failuer or timeout.Where: The Status indicates Q-query, A-abort, w-wait, p-precommit, C-commit)

From the status change diagram, we can see from the dotted line (those F, t are failuer or timeout --If the node is in the P state (precommit) when the F/T problem occurs, the advantage of Three-segment commit is that, the three-segment commit can directly change the status to the C state (COMMIT), while the two-segment commit is overwhelmed..

In fact, three-segment commit is a complicated task, which is quite difficult to implement and has some problems.

Here, I believe you have many problems. You must be thinking about various failure scenarios in 2 PC/3 PC,You will find that timeout is a very difficult task to handle, because the timeout on the network often leaves you with nothing to do, and you do not know whether the other party has done it or not. So you have a good state machine, because timeout becomes a decoration..

A network service has three statuses: 1) Success, 2) failure, 3) Timeout. The third is definitely a nightmare, especially when you need to maintain the status..

Two generals problem (two generals)

The problem with two generals problem is that of a thinking experiment: Two troops, one of which is headed by a general, are preparing to attack a city that has built fortifications. Both of these troops are stationed near the city, occupying one hill. A valley separates the two mountains, and the only communication method between the two generals is to send their respective messengers to and from both sides of the valley. Unfortunately, the Valley has been occupied by the defenders of the city, and there is a possibility that any messenger sent through the valley will be arrested. Please note that, although the two generals have reached a consensus on the attack in the city, they did not reach a consensus on the attack time before they occupied the mountain positions. The two generals must allow their troops to attack the city at the same time to succeed. Therefore, they must communicate with each other to determine a time to attack and agree to the attack at that time. If only one general performs an attack, this would be a catastrophic failure. This thinking experiment involves thinking about how they do it. The following are our ideas:

1) the first general first sent a message "Let's start attacking at nine o'clock A.M ". However, once the messenger is dispatched, it is unknown whether the first general has passed the valley. Any uncertainty will make the first general hesitate to attack, because if the second general cannot launch an attack at the same time, the city's garrison will repel the attack of his army, as a result, his military confrontation was destroyed.

2) knowing this, the second general needs to send a confirmation record: "I have received your email and will attack ." However, what if the sender with the confirmation message is caught? So the second general will hesitate to confirm whether the message will arrive.

3) it seems that we have to send another confirmation message to the first general-"I have received your confirmation ". However, what if the messenger is caught?

4) in this way, do we need the second general to send a message "confirm to receive your confirmation.

Therefore, you will find that this event quickly develops into no matter how many confirmation messages are sent, there is no way to ensure that the two generals are confident that their messenger is not captured by the enemy.

This problem is unsolved.. The two general problems and their unsolved proofs are first identified by E. a. akkoyunlu, K. ekanadham and R. v. huber published the article "limitation and compromise Network Communication Design" in 1975, and described the communication between two gangs in section 73rd of this article. In 465th, Jim Gray named the two generals paradox in his book database operating system considerations (from page 1. This reference has been widely mentioned as a source of definitions and unsolvable proofs of the two generals.

This experiment is intended to clarify the potential and design challenges of trying to coordinate an action through communication established on an unreliable connection.

In terms of engineering, a practical solution to the problem of two generals is to use a solution that can withstand the unreliable communication channels and does not try to eliminate the reliability, however, we need to reduce reliability to an acceptable level. For example, the first general has discharged 100 couriers and is expected to be at least likely to be arrested. In this case, no matter whether the second general will attack or receive any news, the first general will attack. In addition, the first general can send a message stream, while the second general can send a confirmation message for each message, so that if each message is received, the two generals will feel better. However, we can see from the proof that neither of them is sure that this attack can be coordinated. They do not have algorithms available (for example, attacks are triggered when more than four messages are received) to prevent attacks by only one party. In addition, the first general can also number each message, saying that this is the first, the second ...... Until n. This method allows the second general to know how reliable the communication channel is, and return a suitable number of messages to ensure that the last message is received. If the channel is reliable, you only need one message, and the rest will not be of any help. The probability of loss of the last message is equal to that of the first message.

The problem of the two generals can be extended to a more abnormal one.Byzantine generals Problem)The background of the story is as follows: Byzantine is now the capital of the Eastern Roman Empire in Istanbul, Turkey. At that time, because of the vast territory of the Byzantine Roman Empire, for the purpose of defense, each army was separated far away, and the generals and generals had to rely on message transfer. During the war, all the generals in the Byzantine army had to reach an agreement to decide whether they had a chance to win to attack the enemy camp. However, the army may have traitors and enemy spies who disrupt or influence the decision-making process. At this time, how the other loyal generals reach an agreement without the influence of the traitor, even if it is known that some Members have turned against them? This is the question of the Byzantine general.

Paxos Algorithm

The descriptions of various paxos algorithms on Wikipedia are very detailed. You can check them out.

The paxos algorithm solves the problem of how to reach an agreement on a value in a distributed system that may encounter the above exceptions, so as to ensure that no matter any of the above exceptions occurs, the consistency of resolutions will not be damaged. A typical scenario is that, in a distributed database system, if the initial status of each node is consistent and each node executes the same operation sequence, they can finally get a consistent state. To ensure that each node executes the same command sequence column, you need to execute a "consistency algorithm" on each command to ensure that the commands seen by each node are consistent. A general consistency algorithm can be used in many scenarios and is an important issue in distributed computing. Since 1980s, the research on consistency algorithms has not been stopped.

Notes: Paxos is a consistent algorithm based on message transmission proposed by Leslie Lamport, or "La" in latex, which is now at Microsoft Research Institute in 1990. Since the algorithm was hard to understand and did not attract people's attention at first, Lamport was re-published to ACM transactions on computer systems (the Part-time Parliament) eight years later in 1998 ). Even so, the paxos algorithm still did not receive much attention. In 2001, Lamport felt that his peers could not accept his sense of humor, so he restated it in an easy-to-accept method (paxos made simple ). It can be seen that Lamport has a special liking for the paxos algorithm. In recent years, paxos has been widely used to prove its importance in Distributed consistency algorithms. In 2006, three articles by Google began to show the clues of "Cloud". The chubby lock service used paxos as the consistency Algorithm in chubby cell, and the popularity of paxos went viral. (Lamport himself described in his blog how he published this algorithm nine years ago and later)

Note: In Amazon's AWS, all cloud services are implemented based on an alf (async lock framework) framework. This alf uses the paxos algorithm. When I was watching an internal shared video on Amazon, the designer told me in the internal principle talk that he referred to the zookeeper method, but he implemented this algorithm in another way that is easier to read than zookeeper.

To put it simply, paxos aims to make the entire cluster node agree on a change to a value. The paxos algorithm is basically a democratic election algorithm-most of the decisions will be a unified decision of the entire set group. Any node can propose a proposal to modify a data. whether the proposal is passed depends on whether more than half of the nodes in the cluster agree to the proposal (so the paxos algorithm requires the nodes in the cluster to be singular ).

This algorithm has two stages (assuming there are three nodes: A, B, C ):

Phase 1: Prepare stage

A sends the request prepare request for modification to all nodes A, B, and C. Note that the paxos algorithm has a sequence number (you can think of it as a proposal number, which is constantly increasing and unique, that is, a and B cannot have the same proposal number ), this proposal number will be issued together with the modification request. Any end point in the "prepare stage" will reject the request whose value is smaller than the current proposal number. Therefore, when applying for a modification request from all nodes, node A needs to include a proposal number. The larger the proposal number, the larger the proposal number.

If the number N of the proposal received by the receiving node is greater than the number of the proposal sent by other nodes, the node will respond to Yes (the latest approved proposal number on this node ), and ensure that no other <n proposals are received. In this way, the node always promises the latest proposal in the prepare stage.

Optimization: In the above prepare process, if any node finds a proposal with a higher number, you need to notify the publisher to suspend the proposal.

Stage 2: accept stage

If requester A receives yes from more than half of the nodes, then he will send an accept request to all the nodes (similarly, the proposal number N is required). If there are no more than half of the requests, then an error is returned.

After receiving the accept request, if n is the largest for the receiving node, it modifies the value, if you find that you have a larger proposal number, the node rejects modification.

We can see that this seems to be an optimization of "Two-segment commit. In fact,2 pcs/3 pcs are all defective versions of distributed consistency algorithms. Mike Burrows, author of Google chubby, said that there is only one consistent algorithm in the world, namely paxos, and other algorithms are defective.

We can also see that the modification proposal for the same value at different nodes is not problematic even if it is received in disorder on the receiver side.

For some examples, you can take a look at the "paxos example" section in Wikipedia. I will not talk about it here. For some exception examples in the paxos algorithm, you can deduce them by yourself. You will find that as long as more than half of the nodes survive, there is no problem.

Speaking of this, since Lamport published the paxos Algorithm in 1998, various paxos improvements have never been stopped. Among them, the biggest action is the fast paxos published in 2005. Regardless of the improvements, the focus remains on balancing message latency with performance and throughput. To easily distinguish the two, the former is classic paxos, and the latter is fast paxos.

Summary from: Google App Engine Co-founder Ryan Barrett's speech on Google I/O in 2009, transaction messaging SS datacenter (Video: http://www.youtube.com/watch? V = srogpxecblk)

As mentioned above, redundant data must be written in multiple copies to ensure high data availability. Writing multiple copies will bring about consistency, and the consistency will bring about performance problems. We can see that we basically cannot make all items green. This is the famous cap Theory: consistency, availability, and partition tolerance, you only need two of them.

Nwr Model

Finally, I would like to mention the Amazon dynamo nwr model. This nwr model gives the CAP option to the user so that the user can choose which two of your cap.

The so-called nwr model. N indicates n backups, W indicates that at least W copies must be written before they are considered successful, and r indicates that at least R backups are read.W + r> N. Because W + r> N, so what does R> N-W mean? That is, the number of read parts must be greater than the total number of standby parts minus the multiple that ensures the write success.

That is to say, at least one latest version is read each time. So as not to read an old data. When we need a highly writable environment, we can configure W = 1 if n = 3 then r = 3. At this time, as long as any node is successfully written, it is considered successful, but data must be read from all nodes during reading. If we require high Read efficiency, we can configure W = n r = 1. At this time, if any node is successfully read, it is considered successful, but it must be written to all three nodes before it is considered successful.

Some settings of the nwr model may cause dirty data, because it is obviously not a strongly consistent thing like paxos, so, each read/write operation may not be performed on the same node, so some data on the node is not the latest version, but the latest operation is performed.

Therefore, Amazon dynamo introduced the design of the data version. That is to say, if the version of the read data is V1, after you compress the data, you will find that the data version has been updated to V2, then the server will reject you. The version is like an optimistic lock.

However, for the Distributed and nwr models, the version also has a nightmare-that is, the version rush problem. For example, we set n = 3 W = 1, if a value is accepted on node A, the version is changed from V1-> V2, but it has not been synchronized to Node B (asynchronous, W = 1, if a copy is successfully written, it is still V1 on Node B. At this time, Node B receives a write request and, in principle, it needs to be rejected, however, on the one hand, he does not know that other nodes have been updated to V2, and on the other hand, he cannot refuse to write a point because w = 1. As a result, there was a serious version conflict.

Amazon's dynamo cleverly avoids version conflicts-the version rush is handled by the user.

So Dynamo introduced vector clock (vector clock ?!) This design. This design allows each node to record its own version information, that is, for the same data, two things need to be recorded: 1) who updated me, 2) What is my version number.

Next, let's look at an operation sequence:

1) A write request is processed by node A for the first time. Node A adds a version (A, 1 ). We can record the data at this time as D1 (A, 1 ). Then another request for the same key is still processed by a, so there is D2 (A, 2 ). At this time, D2 can overwrite D1 without conflict.

2) Now let's assume that D2 is transmitted to all nodes (B and c). The data received by B and C is not produced by the customer, but copied by others, therefore, they do not generate new version information, so the data held by B and C is still D2 (A, 2 ). The data on A, B, and C and their version numbers are the same.

3) if we have a new write request to Node B, then Node B generates data D3 (A, 2; B, 1), which means: the global version of Data D is 3, A is upgraded to two, and B is upgraded once. Isn't this the so-called code version log?

4) if D3 is not transmitted to C when another request is processed by C, the data on the C node is D4 (A, 2; C, 1 ).

5) Well, the most exciting thing is coming: if there is a read request at this time, we should remember that our w = 1 then r = n = 3, so R will read from all three nodes. At this time, he will read three versions:

• Node A: D2 (A, 2)
• B node: D3 (A, 2; B, 1 );
• C node: D4 (A, 2; C, 1)

6) at this time, it can be determined that D2 is already an old version (already included in D3/D4) and can be discarded.

7) But D3 and D4 are obviously version conflicts. Therefore, the caller must handle version conflicts. Just like source code version management.

Obviously, the above dynamo configuration uses a and P in CAP.

I am pushing everyone to read this paper: Dynamo: Amazon's highly available key-value store. If the English language is painful, you can see the translation)

Original article: http://coolshell.cn/articles/10910.html

Reference http://zh.wikipedia.org/zh/Paxos%E7% AE %97%E6%B3%95#.E5. AE .9E.E4.BE.8B

Http://baike.baidu.com/view/8438269.htm? Fr = Aladdin