Hadoop Learning 13--zookeeper Related

Zookeeper to ensure synchronization between servers, the protocol that implements synchronization is the Zab protocol. There are two modes of this Protocol: Recovery mode (select Master) and broadcast mode (synchronous).

Enter recovery mode when the service starts or leader crashes. After the election is successful and most servers have finished synchronizing with the leader state (N+1 in 2n+1), the recovery mode is over.

State synchronization ensures that the leader and server have the same system state. To ensure the sequential consistency of transactions, zookeeper uses an incremented transaction ID number (ZXID) to identify transactions. All the proposals (proposal) were added to the ZXID when they were presented. The implementation of ZXID is a 64-bit number, it is 32 bits high is the epoch used to identify whether the leader relationship changes, each time a leader is chosen, it will have a new epoch, marking the current period of the reign of the leader. The low 32 bits are used to increment the count.

Synchronization process
After selecting leader, ZK enters the state synchronization process.
1. Leader wait for server connection;
2. Follower connection leader, the largest Zxid sent to leader;
3. Leader the synchronization point according to the zxid of follower;
4. After completing the synchronization notification follower has become uptodate status;
4 O Follower receives the uptodate message, it can re-accept the client's request for service.

Leader has three main functions:
1. Recover data;
2. Maintain heartbeat with learner, receive learner request and Judge learner request message type;
3. The learner message types are mainly ping messages, request messages, ACK messages, revalidate messages, and different processing depending on the message type.
The PING message refers to the heartbeat information of the learner, the request message is the proposed information sent by follower, including write requests and synchronization requests, the ACK message is follower's reply to the proposal, more than half of the follower passed, then commit the proposal The revalidate message is used to extend the session's effective time.

Follower has four main functions:
1. Send a request to the leader (Ping message, request message, ACK message, revalidate message);
2. Receive leader messages and process them;
3. Receive client's request, if write a request, send to leader to vote;
4. Returns the client result.
The follower message loop handles the following messages from leader:
1. Ping message: Heartbeat message;
2. Proposal NEWS: Leader initiated the proposal, request follower vote;
3. Commit message: Information about the latest proposal on the server side;
4. UpToDate message: Indicates that synchronization is complete;
5. Revalidate message: According to Leader's revalidate results, close the session to be revalidate or allow it to accept messages;
6. Sync message: Returns the sync result to the client, originally initiated by the client, to force the latest update.

The above content is quoted from this blog, which has a flowchart:

Http://www.cnblogs.com/kunpengit/p/4045334.html

First, the configuration item explanation:

Zoo.cfg=>

initLimit=10

syncLimit= 5 tickTime= 2000 dataDir=E:/zookeeper/zookeeper- 3.4 . 5 /dataDatalogdir= ... clientPort= 2181 leaderserves=noglobaloutstandinglimit=1000

server.1=slave1:2887:3887
server.2=slave2:2887:3887
server.3=slave3:2887:3887

Explain:

Initlimit: The maximum number of heartbeats (ticktime) that can be tolerated in the initial connection between the follower server (F) in the cluster and the leader Server (L). f at startup, all the latest data is synchronized from the leader, and then the starting state of the external service can be determined. L allow this work to be done within this time frame. If the data volume of the ZK cluster is very large, when the F starts, the synchronization time will be very long, the parameter needs to be adjusted accordingly.

Synclimit: The maximum number of heartbeats (ticktime) that can be tolerated between the follower server in the cluster and the leader server between requests and responses. During the operation of the cluster, L is responsible for communicating with all machines, such as detecting whether the machine is in stock through the heartbeat. If the heartbeat packet does not receive a response within this time frame, the machine will be discarded. This value should not be set too large.

Ticktime: Heartbeat time, the client in the server to retain the session, the minimum session expiration time, is the ticktime twice times.

DataDir: A database snapshot that is used to store in memory.

LogDir:

Leaderserves: By default, the leader machine receives read and write requests from clients. If set to No, the machine will focus on the coordination of the urgent mass machine. This will improve the performance of the write operation.

ClientPort: Listens for the port number of the client connection.

Globaloutstandinglimit: Maximum number of request stacks. The default is 1000. While ZK is running, the server is still allowing clients to submit requests to the server to improve throughput performance, although it has no idle time to handle more client requests. Of course, in order to prevent server memory overflow, this request backlog number still need to be under the limit.

1 in Server.1 corresponds to the value in the myID file in the DataDir directory configured above

Two ports: The first is the port between F and L communication (data synchronization and other communication), and the second is used for leader voting communication in the election.

Extended:

Preallocsize: Pre-open disk space for subsequent writes to the transaction log. The default is 64M, and each transaction log size is 64M. If ZK has a large snapshot frequency, it is advisable to reduce this parameter appropriately.

Snapcount: After each snapcount transaction log output, a snapshot (snapshot) is triggered, and ZK generates a snapshot.* file while creating a new transaction log file log.*. The default is 100000. (In a real code implementation, some random number processing is performed to prevent all servers from taking snapshots at the same time to affect performance)

Tracefile: Log that is used to log all requests, which can be used during general debugging, but is not recommended for production environments and can seriously affect performance.

autopurge.purgeinterval:3.4.0 and later versions, ZK provides the ability to automatically clean up transaction logs and snapshot files, which specifies the cleanup frequency, in hours, a 1 or larger integer, and the default is 0, which means no automatic cleanup is turned on.

Autopurge.snapretaincount: This parameter is used in conjunction with the above parameter, which specifies the number of files that need to be preserved. The default is to keep 3.

Second, the Order:

Start and restart

Bin/zkserver.sh Start/restart

View status

bin/zkserver.sh status

There should be only one leader in the cluster, the rest are follower and looking (the current server does not know who leader is searching for)

To view detailed configuration information for a node:

echo conf | NC slave1 2181

To view the current performance and connection client list for a node:

echo Stat | NC slave1 2181

Simplified version of the above command:

Echo Cons |NC slave1 2181 simply lists the information for the client currently connected to the server

List the details of the current machine environment:

echo reqs |nc slave1 2181

List Watch Details:

echo wchs |nc slave1 2181

List server Watch information by session

echo WCHC |nc slave1 2181

List server watch information by path

echo WCHP |nc slave1 2181

List unprocessed sessions and temporary nodes:

echo Dump |NC slave1 2181

List all outstanding requests:

echo Cons |nc slave1 2181

Another command to turn off the server:

Echo Kill | NC salve1 2181

Open Client

Bin/zkcli.sh-server slave1:2181

List nodes: LS/

Lists node and version information (additions and deletions are updated), number of nodes, etc.: LS2/

Create node: Create/testzk Wordwordword followed by the content string that was deposited

View: Get/testzk including content, version, etc.

Set content string: Set/testzk sssss after execution, replace the

Delete Znode Delete/testzk

Quit client: Quit

Hadoop Learning 13--zookeeper Related