Principle of failure detection in Cassandra

the principle of failure detection in Cassandra Blog Categories:Cassandra Cloud Cassandra Source failure detection phi failure detection

Cassandra principle of failure detection in

first, traditional failure detection and its deficiencies

Traditional failure detection methods

In a distributed system, the Heartbeat (Heartbeat) is often used to detect the health of the server, but in theory, the heartbeat can not really detect whether the other side is crash, the main difficulty is that can not really distinguish between the other is down or "slow". The traditional detection method is to set a timeout time t, as long as within T received the other side of the heartbeat packet is considered the other side of the downtime, the method is simple and rough, but widely used.

defects in traditional error detection

As noted above, in the traditional mode, the target host has a heartbeat per interval of t seconds, and the receiver uses the timeout time t (t<t) to determine whether the target is down, the receiver must first know the target's heartbeat rule (interval of t) to correctly set a timeout T, and T's choice depends on the current network condition, The target host's processing ability and so on many uncertainties, therefore in the practice often will pass the test or the estimate way to assign a upper limit value for T. If the upper limit is set too large, it will lead to the judgment of "slow", but it will increase the correctness of judgment, too small, will improve the efficiency of judgment, but will increase the likelihood of miscalculation. However, the following scenarios do not use the traditional detection method:

1. Gossip communication

However, in practical applications, such as gossip communication applications, because of random communication, there is no regular heartbeat between two servers, so it is difficult to find a suitable timeout time t, unless the T set a very large, but the detection process will be "slow" unbearable.

2. Network Load Dynamic Change

There is also a situation, as the network load increased, the server heartbeat can receive more than the top value T, but when the net pressure decreases, the heartbeat receive time is less than T, if the same as the same t to reflect the heartbeat status, it will cause the judgment "slow" or misjudged.

3. Heartbeat detection and the separation of results

Not every application needs to know the result of a target host being down or not (true/false), that is, there are many applications that need to explain their heartbeat results to take different processing actions. For example, if the target host does not have a heartbeat within 3s, apply a to the downtime and try again, while application B interprets the target as "inactive" and requires delegating tasks to other servers. In other words, whether the target host is "down" should be determined by the business logic rather than simply by a timeout T, which requires separating the heartbeat detection process from the interpretation of the result, thus providing better flexibility for the application.

two Gossiper used in the Φ Failure Detection Method

Proven by the classic thesis of failure detection, the Phi accrual failure Detector (http://vsedach.googlepages.com/HDY04.pdf), in a distributed environment, the heartbeat statistics for the host, Depending on the experience of past heartbeat intervals, the following methods can be used to determine whether a host is down or not.

1. Given a valve value Φ

2. Record each heartbeat interval within a certain period of time

3. The probability of an exponential distribution (exponential distribution) for the interval value of the heartbeat:

P = E ^ ( -1 * (now-lasttimestamp)/mean) (E is logarithmic 2.71828...,mean for this time interval average)

It says that since the last count, the heartbeat arrival time will exceed the probability of now-lasttimestamp

4. Calculation of φ=-log10 P

5. When φ> Φ , you can assume that the host is down.

Of course, this may be a miscarriage of error, the possibility of miscalculation as follows:

Φ = 1, 1%

Φ = 2, 0.1%

Φ = 3, 0.01%

......

This shows that when Φ = 8 o'clock, the misjudgment rate is already very small. Φ = 8 is used by default in Cassandra.

Here is a Java implementation of the Phi failure detection algorithm. The implementation in Cassandra is similar to this.

/** Java Demo for PHI failure detector */ import Java.util.ArrayDeque; import java.util.Iterator; import Java.util.concurrent.locks.Lock; import Java.util.concurrent.locks.ReentrantLock; Public class Phiaccrualfailuredetector { Private Static Final int samplewindowsize = 1000; Private Static int phisuspectthreshold = 8; Private Samplingwindow Simpleingwindow = new Samplingwindow (samplewindowsize); Public Phiaccrualfailuredetector () { } Public void addsample () { Simpleingwindow.add (System.currenttimemillis ()); } Public void addsample (double sample) { Simpleingwindow.add (sample); } Public void interpret () { double phi = Simpleingwindow.phi (System.currenttimemillis ()); System.out.println ("phi =" + phi); if (Phi > Phisuspectthreshold) { System.out.println ("We are assuming the moniored machine is down!"); } Else { System.out.println ("We are assuming the moniored machine is still running!"); } } /** * @param args * The command line arguments */ Public Static void Main (string[] args) { Phiaccrualfailuredetector pafd = new phiaccrualfailuredetector (); I-try with Phi < Phisuspectthreshold for (int i = 0; i < i++) { Pafd.addsample (); Try { Thread.Sleep (10L); catch (Interruptedexception ex) { No op }