一致性演算法探尋（擴充版）8

標籤：

6 Cluster membership changes

Up until now we have assumed that the cluster configuration (the set of servers participating in the consensus algorithm) is fixed. In practice, it will occasionally be necessary to change the configuration, for example to replace servers when they fail or to change the degree of replication. Although this can be done by taking the entire cluster off-line, updating configuration files, and then restarting the cluster, this would leave the cluster unavailable during the changeover. In addition, if there are any manual steps, they risk operator error. In order to avoid these issues, we decided to automate configuration changes and incorporate them into the Raft consensus algorithm.

For the configuration change mechanism to be safe, there must be no point during the transition where it is possible for two leaders to be elected for the same term. Unfortunately, any approach where servers switch directly from the old configuration to the new configuration is unsafe. It isn’t possible to atomically switch all of the servers at once, so the cluster can potentially split into two independent majorities during the transition (see Figure 10).

In order to ensure safety, configuration changes must use a two-phase approach. There are a variety of ways to implement the two phases. For example, some systems (e.g., [22]) use the first phase to disable the old configuration so it cannot process client requests; then the second phase enables the new configuration. In Raft the cluster first switches to a transitional configuration we call joint consensus; once the joint consensus has been committed, the system then transitions to the new configuration. The joint consensus combines both the old and new configurations:

• Log entries are replicated to all servers in both configurations.

• Any server from either configuration may serve as leader.

• Agreement (for elections and entry commitment) requires separate majorities from both the old and new configurations.
  

The joint consensus allows individual servers to transition between configurations at different times without compromising safety. Furthermore, joint consensus allows the cluster to continue servicing client requests throughout the configuration change.

Cluster configurations are stored and communicated using special entries in the replicated log; Figure 11 illustrates the configuration change process. When the leader receives a request to change the configuration from Cold to Cnew, it stores the configuration for joint consensus (Cold,new in the figure) as a log entry and replicates that entry using the mechanisms described previously. Once a given server adds the new configuration entry to its log, it uses that configuration for all future decisions (a server always uses the latest configuration in its log, regardless of whether the entry is committed). This means that the leader will use the rules of Cold,new to determine when the log entry for Cold,new is committed. If the leader crashes, a new leader may be chosen under either Cold or Cold,new, depending on whether the winning candidate has received Cold,new. In any case, Cnew cannot make unilateral decisions during this period.

Once Cold,new has been committed, neither Cold nor Cnew can make decisions without approval of the other, and the Leader Completeness Property ensures that only servers with the Cold,new log entry can be elected as leader. It is now safe for the leader to create a log entry describing Cnew and replicate it to the cluster. Again, this configuration will take effect on each server as soon as it is seen. When the new configuration has been committed under the rules of Cnew, the old configuration is irrelevant and servers not in the new configuration can be shut down. As shown in Figure 11, there is no time when Cold and Cnew can both make unilateral decisions; this guarantees safety.

There are three more issues to address for reconfiguration. The first issue is that new servers may not initially store any log entries. If they are added to the cluster in this state, it could take quite a while for them to catch up, during which time it might not be possible to commit new log entries. In order to avoid availability gaps, Raft introduces an additional phase before the configuration change, in which the new servers join the cluster as non-voting members (the leader replicates log entries to them, but they are not considered for majorities). Once the new servers have caught up with the rest of the cluster, the reconfiguration can proceed as described above.  

The second issue is that the cluster leader may not be part of the new configuration. In this case, the leader steps down (returns to follower state) once it has committed the Cnew log entry. This means that there will be a period of time (while it is committing Cnew) when the leader is managing a cluster that does not include itself; it replicates log entries but does not count itself in majorities. The leader transition occurs when Cnew is committed because this is the first point when the new configuration can operate independently (it will always be possible to choose a leader from Cnew). Before this point, it may be the case that only a server from Cold can be elected leader.

The third issue is that removed servers (those not in Cnew) can disrupt the cluster. These servers will not receive heartbeats, so they will time out and start new elections. They will then send RequestVote RPCs with new term numbers, and this will cause the current leader to revert to follower state. A new leader will eventually be elected, but the removed servers will time out again and the process will repeat, resulting in poor availability. 

To prevent this problem, servers disregard RequestVote RPCs when they believe a current leader exists. Specifically, if a server receives a RequestVote RPC within the minimum election timeout of hearing from a current leader, it does not update its term or grant its vote. This does not affect normal elections, where each server waits at least a minimum election timeout before starting an election. However, it helps avoid disruptions from removed servers: if a leader is able to get heartbeats to its cluster, then it will not be deposed by larger term numbers. 

6 叢集成員關係的改變

到目前為止，我們假定叢集配置（一組伺服器參與一致性演算法）是固定的。實際上，偶爾是需要改變更配置置的，例如當伺服器掛了後改變複製程度時取代他們。雖然這可以通過將整個叢集離線，更新設定檔，然後重啟叢集來完成，但這將導致在叢集轉換期間系統不可用。另外，任何步驟中錯誤操作的風險大增。

為了保證配置變更機制是安全的，如果它可能導致在轉換期間有兩個leader在同一term中當選，那就變得沒有任何意義了。不幸的是，任何伺服器直接從舊配置切換到新配置的策略都是不安全的。一次性原子切換所有伺服器是不可能的，所以叢集可以在切換期間分為兩個獨立的部分（見圖10）。

為了確保安全，配置切換必須採取二階段的策略。有很多方法來實現二階段。例如，一些系統（如[22]）採用一階段禁用舊配置，因此它不能處理用戶端的請求；然後二階段採用新配置。Raft中，叢集首先切換到我們叫做joint consensus的過渡配置；一旦joint consensus提交完成，系統就過渡到了新配置。joint consensus結合了舊的和新的配置：

• 兩種配置下都將日誌複製到所有伺服器。

• 任何一種配置下的任何一台伺服器都可以充當leader。

• 協議（選舉和條目提交）不管舊的還是新的配置都需要分離大部分伺服器。

joint consensus運行個別的伺服器在不同時間不影響安全性的前提下切換配置。此外，joint consensus運行叢集在配置更改期間繼續服務用戶端請求。

叢集配置使用特殊的已複製日誌條目儲存和通訊；圖11展示了配置更新的過程。當leader接收到配置從Cold到Cnew切換的請求時，作為一個條目儲存joint consensus的配置（圖中的Cold,new），並用之前描述的機制進行複製。一旦給定的伺服器增加了新配置條目到它的日誌中，它使用未來決策的配置（伺服器總是在它的日誌中使用最新的配置，無論條目是否被提交）。這意味著leader將會採用Cold,new的規則來決定Cold,new的日誌條目何時被提交。如果leader崩潰了，一個新的leader將可能在Cold或Cold,new下當選，而這取決於獲勝的candidate是否接收到了Cold,new。任何情況下，Cnew不能在此期間單方面做決定。

一旦Cold,new被提交，無論Cold還是Cnew都無法在另一方沒同意的情況下做出決定了，the Leader Completeness Property確保了只有Cold,new下的伺服器日誌條目可以當選成為leader。現在對於leader建立新的日誌條目描述Cnew和將之複製到叢集是安全的。再次，配置就能起到所見即所得 (WYSIWYG)的效果了。當新的配置在Cnew規則下被提交，Cold配置就無關緊要了，並且不在新配置下的服務就會可以關閉。11所示，不存在Cold和Cnew同時單方面決定的時候；這保證了安全性。

重新設定還有3個需要解決的問題。第一個問題是新的伺服器開始可能沒存任何日誌條目。如果他們是這種狀態下接入叢集的，可能就要花好一會兒來追上進度了，在這期間它就可能無法提交新的日誌條目了。為了避免可能性差距，Raft在配置更新前引入了額外的一個階段，新伺服器作為非選民加入叢集（leader將日誌條目複製給他們，但他們沒有選舉權）。一旦新伺服器趕上了其他伺服器的進度，重新設定就可以如上所示的進行了。

第二個問題是叢集的leader可能不是新配置的那部分。leader一旦提交完Cnew日誌條目就下台（返回follower狀態）。這意味著有一段時間（它在提交Cnew條目時）當leader在管理叢集時不包含自己；它複製日誌條目，但不將自己計算到大多數中。當Cnew因為新配置可以獨立地操作而被提交（總是從Cnew中選取leader），leader才進行過渡。在這之前，可能會有只有一個Cold的伺服器可以當選leader。

第三個問題是去除伺服器（不在Cnew的那些）可能破壞叢集。這些伺服器將接收不到心跳，他們就會逾時並啟動一次新的選舉。他們將會發送新的term值的RequestVote RPC，這會導致現任leader恢複follower狀態。一個新的leader當選，但去除伺服器會再次逾時，並且這個過程會再一次重複，導致可用性變差。

一致性演算法探尋（擴充版）8