Consistency Algorithm Quest (Extended version) 11

9 Implementation and evaluation

We have the implemented Raft as part of a replicated state machine This stores configuration information for Ramcloud [+] and Assists in failover of the Ramcloud coordinator. The Raft implementation contains roughly lines of C + + code, not including tests, comments, or blank lines. The source code is freely available [23]. There is also about independent Third-party open source implementations [the] of Raft in various stages of development, Based on drafts of this paper. Also, various companies is deploying raft-based systems [34].

The remainder of this section evaluates Raft using three criteria:understandability, correctness, and performance.

9.1 understandability

To measure Raft ' s understandability relative to Paxos, we conducted an experimental study using Upper-level undergrad Uate and graduate students in an advanced Operating Systems course at Stanford University and a distributed Computing cour Se at u.c. Berkeley. We recorded a video lecture of Raft and another of Paxos, and created corresponding quizzes. The Raft lecture covered the content of this paper except for log compaction; The Paxos lecture covered enough material to create a equivalent replicated state machine, including Single-decree Paxos, Multi-decree Paxos, reconfiguration, and a few optimizations needed in practice (such as leader election). The quizzes tested basic understanding of the algorithms and also required students to reason about corner cases. Each student watched one video, took the corresponding quiz, watched the second video, and took the second quiz. About half of the participants do the Paxos portion first and the other half do the Raft portion first in Order to account for both individual differences in performance and experience gained from the first portion of the study . We compared participants ' scores on each quiz to determine whether participants showed a better understanding of Raft.

We tried to make the comparison between Paxos and Raft as fair as possible. The experiment favored Paxos in the ways:15 of the participants reported have some prior experience with Paxos, and The Paxos video is 14% longer than the Raft video. As summarized in Table 1, we have taken steps to mitigate potential sources of bias. All of our materials is available for review [28, 31].

On average, participants scored 4.9 points higher on the Raft quiz than on the Paxos quiz (out of a possible points, th E mean Raft score was 25.7 and the mean Paxos score was 20.8); Figure shows their individual scores. A paired t-test states, with 95% confidence, the true distribution of Raft scores have a mean at least 2.5 points larg Er than the true distribution of Paxos scores.

We also created a linear regression model that predicts a new student ' s quiz scores based on three Factors:which Quiz They took, their degree of prior Paxos experience, and the order in which they learned the algorithms. The model predicts, the choice of quiz produces a 12.5-point difference in favor of Raft. This is significantly higher than the observed difference of 4.9 points, because many of the actual students had prior Pax OS experience, which helped Paxos considerably, whereas it helped Raft slightly less. Curiously, the model also predicts scores 6.3 points lower on Raft for people that has already taken the Paxos quiz; Although we don ' t know why, this does appear to be statistically significant.

We also surveyed participants after their quizzes to see which algorithm they felt would being easier to implement or explain ; These results is shown in Figure 15. An overwhelming majority of participants reported Raft would being easier to implement and explain N). However, these self-reported feelings may is less reliable than participants ' quiz scores, and participants could have B Een biased by knowledge of we hypothesis that Raft are easier to understand.

A detailed discussion of the Raft user study is available at [31].

9.2 Correctness

We have developed a formal specification and a proof of safety for the consensus mechanism described in section 5. The formal specification [to] makes the information summarized in Figure 2 completely precise using the tla+ specification Language [17]. It is about lines long and serves as the subject of the proof. It is also useful in its own for anyone implementing Raft. We have mechanically proven the Log completeness property using the TLA proof system [7]. However, this proof relies in invariants that has not been mechanically checked (for example, we had not proven the type Safety of the specification). Furthermore, we have written an informal proof [+] of the state machine Safety property which are complete (it relies on t He specification alone) and relatively precise (it's about 3500 words long).

9.3 Performance

Raft ' s performance is similar to other consensus algorithms such as Paxos. The most important case for performance was when a established leader is replicating new log entries. Raft achieves this using the minimal number of messages (a single round-trip from the leader to half the cluster). It is also possible to further improve Raft ' s performance. For example, it easily supports batching and pipelining requests for higher throughput and lower latency. Various optimizations has been proposed in the literature for other algorithms; Many of these could is applied to Raft and we leave this to the future work.

We used our Raft implementation to measure the performance of Raft ' s leader election algorithm and answer both questions. First, does the election process converge quickly? Second, what's the minimum downtime that can being achieved after leader crashes?

To measure leader election, we repeatedly crashed The leader of a cluster of five servers and timed what long it to OK to detect the crash and elect a new leader (see Figure 16). To generate a worst-case scenario, the servers in each trial had different log lengths, so some candidates were not Eligib Le to become leader. Furthermore, to encourage split votes, we test script triggered a synchronized broadcast of Heartbeat RPCs from the Leade R before terminating its process (this approximates the behavior of the leader replicating a new log entry prior to Crashi NG). The leader was crashed uniformly randomly within it heartbeat interval, which was half of the minimum election timeout fo R all tests. Thus, the smallest possible downtime is about half of the minimum election timeout.

The top graph in figure shows, a small amount of randomization in the election timeout is enough To av OID split votes in elections. In the absence of randomness, leader election consistently took longer than ten seconds in we tests due to many split vote S. Adding just 5ms of randomness helps significantly, resulting in a median downtime of 287ms. Using more randomness improves worst-case behavior:with 50ms of randomness the worstcase completion time (over the trial s) was 513ms.

The bottom graph in figure shows this downtime can be reduced by reducing the election timeout. With an election timeout of 12–24ms, it takes is 35ms on average to elect a leader (the longest trial took 152ms). However, lowering the timeouts beyond this point violates Raft ' s timing requirement:leaders has difficulty broadcasting Heartbeats before other servers start new elections. This can cause unnecessary leader changes and lower overall system availability. We recommend using a conservative election timeout such as 150–300ms; Such timeouts is unlikely to cause unnecessary leader changes and would still provide good availability.

9 Implementation and evaluation

We have implemented the raft as part of the replication state machine that holds the Ramcloud configuration and the Ramcloud cache failure collaborator information. The raft implementation contains approximately 2000 lines of C + + code, excluding tests, comments, and blank lines. [23] These sources are free of charge. There are about 25 third-party open-source implementations of raft at different stages of development [34], based on this paper. In addition, many companies have deployed raft-based systems.

The remaining chapters will use three criteria to evaluate the raft: clarity, correctness, and performance.

9.1 Easy Accessibility

In order to evaluate raft's understanding with Paxos, we studied advanced Operating systems from Stanford University and studied at the University of California, Berkeley distributed An experiment was conducted by senior students and graduates of the computing course. We recorded a video lecture on Raft and Paxos, and set up a corresponding quiz. Raft Lecture covers the paper except for the journal compression; Paxos lectures cover enough material to create a replication state machine, including Single-decree Paxos,multi-decree Paxos, Reconfiguration and some of the optimizations needed in practice (such as leader elections). The quiz tests the basic understanding of the algorithm and requires students to introduce extreme situations. Each student only watches one video, makes the appropriate quiz, then watches the second video and does the corresponding quiz. In order to reflect the differences in the experience and performance of the two in this study, almost half of the participants first did the Paxos part, and the other half did the raft part for the first time. We compared the scores of each test participant to determine if the participants showed raft better understanding.

We tried to make the contrast between Paxos and raft as fair as possible. There are two types of preference for Paxos in experiments: 15 of the 43 participants have existing experience with some Paxos, and Paxos videos are 14% longer than raft. As shown in table 1, we have taken steps to reduce the deviations from the participating sources. All of our materials can be reviewed [28,31].

On average, participants scored 4.9 points in the raft test, higher than Paxos (converted to 60 points, Raft scored 25.7,paxos score of 20.8), and figure 14 showed their respective scores. Pairing T-test shows that we have 95% confidence that the raft score distribution is at least 2.5 points higher than Paxos.

We also set up a factor-based linear regression model to speculate on a new participant's quiz score: which test they used, their previous Paxos experience, and the order in which they learned the algorithm. The choice of model prediction Test will make 12.5-point people love raft. This is obviously more of a problem than 4.9 points, because in fact many students have Paxos experience before, which has helped Paxos considerably, and it has been very rare to help raft. Oddly, the model also predicts that people who have done Paxos tests have a score of 6.3 points lower than those who only do raft, although we don't know why, but it does appear statistically.

We also investigated what they thought was easier to implement and interpret the algorithms they were exposed to when they finished the tests, as shown in Figure 15. The overwhelming majority of participants indicated that raft was easier to implement and interpret (41 questions in 33). However, this self-reported feeling is less reliable than the participants ' test scores, and these participants may be more understandable and biased in the knowledge we have set at the raft point.

Raft a detailed discussion of user studies in [31].

9.2 Correctness

Consistency Algorithm Quest (Extended version) 11