The trouble of Distributed system------"Designing data-intensive Applications" Reading notes 11

The use of distributed systems and in the single-machine system to deal with the problem is very different, the distributed system has brought greater processing power and storage capacity, but also brought a lot of new "trouble." In this article, we will look at the new challenges that distributed systems bring to us.

1. Failure

When we're using a stand-alone system, it usually works in a fairly predictable way: either it works or it doesn't work.

And when we're using distributed systems, things are different. In a distributed system, some parts of the system may be destroyed in some unpredictable way, even if other parts of the system are working properly. This failure is often uncertain: if you want to do something that involves multiple nodes and networks, you may not even know if a message is successful, because the time it takes for the message to traverse the network is uncertain.

The uncertainty of this kind of failure makes the distributed system become complex and fragile. The larger a system, the more likely it is that its components will fail. In a system with tens of thousands of nodes, something always goes wrong. while the error-handling strategy is simply to give up, a large system can spend a lot of time recovering from a failure rather than doing useful work. so we need the distributed system to tolerate the failed nodes, and still maintain the overall work, the fault-tolerant mechanism built into the software. In other words, a distributed system needs to build a reliable system from an unreliable component.

2. Unreliable networks

A distributed system is a set of network-attached machines. The network is the only way for these machines to communicate, each machine has its own memory and disk, and one machine cannot access the memory or disk of another machine. in a network, a node can send a message to another node, but the network cannot guarantee when it arrives or arrives, so the network is unreliable.

As shown, if you send a request that is not responding, you cannot distinguish between
(a) Loss of request
(b) Remote node failure
(c) The response is lost.
The usual way to handle this problem is to timeout: After a period of time, the sender discards the wait and assumes that the response does not arrive. However, when the time-out occurs, the remote node may have been requested and processed.

Fault detection

The uncertainty of the network makes it difficult to determine whether a node is working. The mechanism of time-out detection is commonly used in distributed systems. if time-out detection is a method of detecting a failure, how long should the timeout be? Unfortunately, there is no simple answer.

A long time-out means waiting for a node to be declared dead. A short timeout will detect a failure more quickly, but the fact that the node does not stop working (for example, due to a node or network overload) incorrectly detects a node failure. If the node is actually alive, and when some operation is performed, the other node is taken over, the operation may eventually be executed two times. and when one node fails, its responsibilities need to be shifted to the other nodes, which puts additional load on the other nodes and the network. If the system is already under high load, premature detection of node failure can make the problem worse. in particular, it may occur that the node is not actually aging, but the response is slow due to overloading, and transferring its load to other nodes can lead to cascading failures.

The current trends in academia and industry are: instead of using a constant-configured timeout, the system can continuously measure the response time and response time jitter, and automatically adjust the time-out to dynamically distribute according to the observed response time. such as Akka's timeout, Cassandra Dynamic detection, TCP timeout retransmission.

3. Unreliable time

In a distributed system, time is a tricky thing, because communication is not instantaneous: It takes time for a message to traverse a network from one machine to another. Messages are always received more often than they are sent, but due to the variable latency in the network, we do not know how much delay will be in the future. It is difficult to determine the logic and sequence of multiple machine processes.

Each machine has its own clock, usually a quartz crystal oscillator. These devices are not completely accurate, so each machine has its own time, which may be slightly faster or slower than other machines. Network protocols with synchronous clocks: the most common mechanism is the Network Time Protocol (NTP), which allows the computer clock to adjust according to the time reported by a group of servers. The server can obtain time from a more precise time source.

Clock:

The UTC time starts with January 1, 1970, and the current time is calculated based on the Gregorian calendar, ignoring leap seconds. The computer clock is usually synchronized with NTP, which means that the timestamp of one machine (ideally) means the same as the timestamp on the other machine.

Monotony of Time:

You can check the value of the clock at a point in time, and then check the clock again. The difference between the two values tells you how much time is spent between the two checks. In a distributed system, measuring time through a monotonic clock (such as timeouts) is usually good because it does not assume a slight error of synchronization between the clocks of different nodes.

Timestamp sort of Event

sequencing events across multiple nodes is a daunting issue. For example, if two clients write to a distributed database, who first arrives? Which one was recently written? as shown in the following:

Write x = 1 timestamp is 42.004 seconds, but write x = 2 timestamp for 42.003 seconds. When node 2 receives both events, it mistakenly concludes that x = 1 is the most recent value, ignoring x=2 writes. The incremental operation of Client B will be lost. This conflict resolution strategy, known as the last writer wins (LWW), causes a node with a lag clock to fail to overwrite the value of a node previously written with a fast clock until the clock skew between the nodes disappears.

So for systems with strict timing requirements, a logic clock is required (for example: Lamport Clock,lanport is really a distributed field of ancient God Ox ~ ~ ~), which is based on the increment counter is one to judge the sequence of events. The logical clock does not measure the time of day or the number of seconds elapsed, only the relative order of events, which is to determine whether an event occurs before or after another event.

4. Unreliable leases

In a distributed system, it is sometimes necessary to ensure that the storage service file is accessible only to one client at a time, because if multiple clients try to write it, the file is corrupted. You need to implement a distributed lock by obtaining a lease from the lock service before accessing the file. But sometimes this lock is not as reliable as we might think, as shown in:

If the client holding the lease is 1 suspended for a reason such as a GC, its lease expires. Another client 2 can obtain a lease and begin writing data to the file. When the paused client 1 returns, it still considers itself to have a valid lease and continues to write the data. This creates a write conflict.

Fence Tokens

We can use fence tokens to make unreliable leases more reliable, as shown in:

The lock server can return a token each time a lease is granted, which is a numeric ID that is incremented each time a lock is granted. Each time a client makes a write request, it must include the current lease token. The storage service will record the written lease token, become a fence, and the old token write will be rejected by the storage service.

Summary:

The biggest challenge for distributed systems is that we need to build a reliable system between unreliable components and complex multi-node interactions, so we need to do more. I have skipped over the Byzantine question, and usually the data system we develop is considered Byzantine safe, and the nodes can be trusted.

The trouble of Distributed system------"Designing data-intensive Applications" Reading notes 11