MIT 6.824:spring LAB2 Training Note

Lab 2:primary/backup Key/value Service

Overview of Lab 2

In this experiment, we will use the Primary/backup replication to provide a fault-tolerant key/value service. In order for all clients and severs to agree which server is primary and which server is backup, we will introduce a master service called Viewservice. Viewservice will monitor which of the available servers is dead and which are alive. If the current primary or backup is dead, Viewservice will choose a server to replace it. The client obtains the current primary by checking the viewservice. Servers works with Viewservice to ensure that at most one primary at any time.

Our Key/value service will be able to replace the failed servers. When a primary fails, Viewservice chooses one from backup as the new primary. When a backup fails or is selected as primary, it becomes backup if a free server,viewservice is available. Primary will send the entire database to the new backup, as well as the contents of the subsequent puts operation to backup, ensuring that the backup Key/value database and primary are the same.

In fact, the primary must send the get and puts operations to backup (if present) and reply to the client until the backup response is received. This prevents two servers from playing the role of primary (a "split brain") at the same time. For example: S1 is Primary,s2 is backup. Viewservice (wrongly) thought S1 dead and promoted S2 to the new primary. But the client still thinks that S1 is primary and sends a operation to it. S1 will send the operation to S2,s2 will reply to an error telling S1 that it is no longer backup (assuming S2 has acquired a new view from Viewservice). So S1 will return a mistake to the client indicating that S1 may no longer be primary (because S2 rejected operation, so a new view must have been formed). After that, the client will ask Viewservice to get the correct primary (S2) and send operation to it.

The failed Key/value server needs to be restarted, but at this point we do not need to copy replicated data (those keys and value). This means that our Key/value server is storing the data in memory instead of on disk. One consequence of storing data only in memory is that if no backup,primary fails and a restart is performed, it will no longer play primary.

RPC is the only way to interact between clients and servers, between different servers, and between different clients. For example, it is not allowed to share go variables or files between different server instances.

The design described above has some limitations on fault tolerance and performance, making it difficult to apply in the real world:

(1), Viewservice is very fragile because it is not backed up

(2), primary, and backup must perform one operation at a time, limiting their performance

(3), recovering server must copy the entire Key/value-pair database from primary, even if it already has nearly the most recent data, which is very slow (for example, possibly due to network problems and thus less than a few minutes of updates).

(4), because servers does not store the Key/value database on disk, it can not tolerate the simultaneous crash of the server (for example, the entire site range of power outages)

(5), if the communication between primary and backup is hindered by a temporary problem, the system has only two remedies: Change view, eliminate the communication barrier backup, or keep trying, either way, if such a problem always happens, performance will not be very good

(6) If primary fails before confirming that it is a primary view, then Viewservice will not be able to continue-----it will spin and will not change view

In subsequent experiments, we will address these limitations through better design and protocols. This experiment will let you know what problems will be solved in the next experiment.

The PRIMARY/BACKUP scheme in this experiment is not based on any known protocol. In fact, this experiment does not specify a complete protocol, and we have to deal with the details ourselves. This experiment in fact and flat Datacenter storage somewhat similar (viewservice like FDS metadata center,primary/backup server like the Tractserver in FDS), But FDS took more effort in performance optimization. The design of this experiment is similar to the replica set in MongoDB, although MongoDB chooses leader by Paxos-like election. For a detailed description of the Primary-backup-like protocal, see the implementation of the chain replication. Chain replication has better performance than the design of this experiment, although its viewservice does not announce the death of a server if it just participates. See Harp and viewstamped Replication for detailed processing of high-performance primary/backup and for refactoring operations on system state after a variety of failures.

Part a:the Viewservice

Viewservice passes through a series of labeled view, each view has a primary and a backup (if any). A view consists of a view number and view's primary and backup severs's identity (network port number). The primary of a view must be the primary or backup of the previous view. This ensures that the state of the key/value can be preserved. Of course there is an exception: when Viewservice is just booting up, it will be able to accept any server as the first primary. Backup in view can be any server other than primary, and it can be without backup if there is no server available. (represented by an empty string, "")

Each key/value server sends a ping RPC to viewservice,viewservice at every other pinginterval to reply to the description of the current view. Ping lets Viewservice know that Key/value server is still alive, notifies the current view of Key/value server, and lets Viewservice know the latest view Key/value server knows. If Viewservice passes Deadpings Pingintervals has not received a ping from the server, Viewservice thinks the server is dead. When a server crashes, it needs to send one or more pings with parameter 0 to Viewservice to tell Viewservice that it crashed.

When (1) Viewservice does not get the latest ping from primary and backup, (2) Primary or backup crashes and restarts, (3) If no backup is currently present and there is an idle server (a server Ping, but it is neither primary nor backup, Viewservice will enter a new view. However, Viewservice must not change the view until the current view's primary confirms that it is working on the current view (by sending a ping with the current view number). When Viewservice still does not receive the current view of the primary for the current view acknowledgment, it cannot change the view even though it thinks primary or backup is dead. Simply put, Viewservice cannot enter view x+1 from view x if it has not received a ping (X) from the primary of view X.

This acknowledge rule prevents viewservice view from more than one Key/value server. If Viewservice can lead any view, then we need a more complex design that will keep the history of the view in Viewservice so that Key/value server can get the old view, and to recycle the old view at the right time. The drawback of this acknowledgment rule is that if primary fails before it confirms that it is a primary view, then Viewservice can no longer change view.

The viewservice part of source code analysis

The viewserver structure is as follows:

Type viewserver struct {mu  sync. Mutex l NET. Listener dead Int32//For Testing rpccount int32  //For testing me string//Your declaration here.}

Src/viewservice/server.go

Func startserver (Me string) *viewserver

(1), first fill in a *viewserver data structure

(2), call RPCs: = RPC. NewServer () and RPCs. Register (VS), registering an RPC server

(3), call L, E: = Net. Listen ("Unix", vs.me), VS.L = l Establish network connection

(4), generate two Goroutine, one to receive RPC request from client and generate Goroutine processing, another goroutine call tick () every pinginterval

The clerk part of source code analysis

The clerk structure is as follows:

The Viewservice clerk lives in the client and maintains a little statetype clerk struct {me  string//Client ' s name (host:port) server string//Viewservice ' s Host:port}

　　

Src/viewservice/client.go

Func Makeclerk (Me string, server string) *clerk

The function simply fills in a *clerk structure and returns.

Src/viewservice/client.go

Func (CK *clerk) Ping (viewnum int) (View, error)

Create the variable, args: = &pingargs{}, and populate it, then call OK: = called (Ck.server, "viewserver.ping", args, &reply) and return to reply. View

The view part of source analysis

The view structure looks like this:

Type View struct {viewnum  int Primary  string Backup   string}

　　

The ping-related structure is as follows:

If Viewnum is zero, the caller are signalling that it is alive and could become backup if Neededtype pingargs struct { Me string//"Host:port" viewnum UINT//caller ' s notion of current View #}

Type pingreply struct {
View View
}

　　

Get-related structures are as follows:

Get (): Fetch the current view, without volunteering to is a server.mostly for clients of p/b service, and for Testingty PE getargs struct {}type getreply struct {view View}

　　

The viewserver would declare a client dead if it misses this many Ping RPCs in a row

Const DEADPINGS = 5

------------------------------------------------------------------------------------------------Test Framework Analysis--------- --------------------------------------------------------------------------------------

1, Src/viewservice/test_test.go

Func Check (t *testing. T, CK *clerk, p string, b string, n UINT)

The function first calls the view, _: = ck. Get () Gets the current view and compares the view's Primary,backup and p, B is equal, and when n is not 0, compares N and view. Viewnum are equal.

Last Call CK. Primary () compare and p are equal.

2, Src/viewservice/test_test.go

Func Test1 (t *testing. T

(1), First call runtime. Gomaxprocs (4), then specify Viewservice's Port,vshost: = Port ("V"), in the format "/var/tmp/824-${uid}/viewserver-${pid}-v"

(2), call vs: = StartServer (vshost) Start Viewservice

(3), call Cki: = Makeclerk (Port ("I"), vshost), i = 1, 2, 3, start 3 server

(4), when Ck1. Primary () is not empty, it is an error, because there should be no Primary at this time.

Test:first primary ...

Every other pinginterval ck1 is called once ck1. Ping (0) operation until the view is returned. Primary Exit for Ck1.me, up to 2 cycles deadpings *

Test:first backup ...

First Call VX, _: = Ck1. Get gets the current view, which is called once per PingInterval ck1 ck1. Ping (1), after which Ck2 calls view, _: = Ck2. Ping (0) operation until the view is returned. When backup is ck2.me, it exits with a maximum loop of deadpings * 2 times.

Test:backup takes over if primary fails ...

First by calling Ck1. Ping (2) Confirm the following view 2. Then call VX, _: = Ck2. Ping (2) Gets the current view of Viewservice, and then once every PingInterval call V, _: = Ck2. Ping (VX. Viewnum), until v. Primary = = Ck2.me and V.backup = = "", up to cycle deadpings * 2 times.

MIT 6.824:spring LAB2 Training Note