Introduction to Zookeeper working principle, installation configuration, tool commands

1 Zookeeper Introduction
Zookeeper is a distributed service framework, which is mainly used to solve some data management problems commonly encountered in distributed applications, such as: Unified Naming Service, State Synchronization service, cluster management, distributed application configuration item management and so on.
2 Zookeeper Basic Concepts
2.1 Characters
There are three main categories of roles in Zookeeper, as shown in the following table:

System Model:

2.2 Design Purpose
1. Final consistency: No matter which server the client connects to, it is the same view that is presented to it, which is the most important performance of zookeeper.
2. Reliability: With simple, robust, good performance, if the message M is accepted to a server, then it will be accepted by all servers.
3. Real-time: zookeeper to ensure that the client will be in a time interval to obtain updates to the server, or server failure information. However, due to network delay and other reasons, zookeeper cannot guarantee that two clients can get the newly updated data at the same time, if you need the latest data, you should call the sync () interface before reading the data.
4. Wait unrelated (Wait-free): Slow or invalid client must not intervene in the fast client request, so that each client can effectively wait.
5. Atomicity: Updates can only succeed or fail with no intermediate state.
6. Sequence: including global order and partial order: Global order is that if the message a on a server is published before message B, on all servers, message A will be published in front of message B; The partial order is that if a message B is published by the same sender after message A, a must precede B.
3 Zookeeper Working principle
The core of Zookeeper is broadcast, a mechanism that guarantees synchronization between the various servers. The protocol that implements this mechanism is called the Zab protocol. The ZAB protocol has two modes, namely the recovery mode (select Master) and broadcast mode (synchronous). When the service is started or after the leader crashes, the Zab enters the recovery mode, and when the leader is elected and most of the servers are synchronized with the leader state, the recovery mode is finished. State synchronization ensures that the leader and server have the same system state. To ensure the sequential consistency of transactions, zookeeper uses an incremented transaction ID number (ZXID) to identify transactions. All the proposals (proposal) were added to the ZXID when they were presented. The implementation of ZXID is a 64-bit number, it is 32 bits high is the epoch used to identify whether the leader relationship changes, each time a leader is chosen, it will have a new epoch, marking the current period of the reign of the leader. The low 32 bits are used to increment the count.
Each server has three states in the process of working:
Looking: Current server does not know who leader is, is searching.
Leading: The current server is an elected leader.
The Following:leader has been elected and the current server is in sync with it.
3.1 Selection Master Process
When leader crashes or leader loses most of the follower, when ZK enters recovery mode, the recovery mode needs to re-elect a new leader, so that all servers are restored to a correct state. ZK's election algorithm has two kinds: one is based on basic Paxos, the other is based on the fast Paxos algorithm. The default election algorithm for the system is fast Paxos.
Basic Paxos Process:
1. The election thread is held by the current server-initiated election thread, whose main function is to count the poll results and select the recommended server;
2. The election thread first initiates an inquiry (including itself) to all servers;
3. After the election thread receives the reply, verifies whether it is an inquiry initiated by itself (verifies that the ZXID is consistent), then obtains the other person's ID (myID), and stores it in the current Query object list, and finally obtains leader related information (ID,ZXID) proposed by the other party. and store this information in the Voting record table of the election;
4. After receiving all the server replies, calculate the ZXID largest server, and set the server related information to the next server to vote;
5. The thread sets the current ZXID maximum server to the current server to recommend leader, if the winning server obtains N/2 + 1 of the server votes, sets the currently recommended leader for the winning server, will set its own state based on the information about the winning server, otherwise, continue the process until leader is elected.
Through process analysis we can conclude that to enable leader to obtain support from most servers, the total number of servers must be odd 2n+1 and the number of surviving servers should not be less than n+1. The above process is repeated after each server startup. In recovery mode, if the server that is just recovering from a crash or just started also recovers data and session information from a disk snapshot, ZK logs the transaction log and periodically snapshots it to facilitate state recovery on recovery.
The specific flowchart for the selected master is as follows:

The Fast Paxos process is an election process in which a server first proposes itself to be a leader to all servers, and when other servers receive the offer, resolves the clash between the epoch and ZXID and accepts the other's proposal, then sends a message to the other party accepting the proposal to complete, Repeat the process, and you will finally be able to elect the leader. The flowchart is as follows:

3.2 Synchronization Process
After selecting leader, ZK enters the state synchronization process.
1. Leader wait for server connection;
2. Follower connection leader, the largest Zxid sent to leader;
3. Leader the synchronization point according to the zxid of follower;
4. After completing the synchronization notification follower has become uptodate status;
5. Follower receives the uptodate message, it can re-accept the client's request for service.
The flowchart is as follows:

3.3 Work Flow
3.3.1Leader Work Flow
Leader has three main functions:
1. Recover data;
2. Maintain heartbeat with learner, receive learner request and Judge learner request message type;
3. The learner message types are mainly ping messages, request messages, ACK messages, revalidate messages, and different processing depending on the message type.
The PING message refers to the heartbeat information of the learner, the request message is the proposed information sent by follower, including write requests and synchronization requests, the ACK message is follower's reply to the proposal, more than half of the follower passed, then commit the proposal The revalidate message is used to extend the session's effective time.
The leader Workflow diagram is as follows:

3.3.2Follower Work Flow
Follower has four main functions:
1. Send a request to the leader (Ping message, request message, ACK message, revalidate message);
2. Receive leader messages and process them;
3. Receive client's request, if write a request, send to leader to vote;
4. Returns the client result.
The follower message loop handles the following messages from leader:
1. Ping message: Heartbeat message;
2. Proposal NEWS: Leader initiated the proposal, request follower vote;
3. Commit message: Information about the latest proposal on the server side;
4. UpToDate message: Indicates that synchronization is complete;
5. Revalidate message: According to Leader's revalidate results, close the session to be revalidate or allow it to accept messages;
6. Sync message: Returns the sync result to the client, originally initiated by the client, to force the latest update.
The follower workflow diagram is as follows:

4 Zookeeper Installation Configuration
Zookeeper's installation mode is divided into three types, namely: Single machine mode (stand-alone), cluster mode and cluster pseudo-
Distribution mode.
4.1 Stand-alone mode
After downloading the zookeeper installation package, unzip to the appropriate directory. Enter the Conf subdirectory under the Zookeeper directory to create the ZOO.CFG:
ticktime=2000
Datadir=/users/apple/zookeeper/data
Datalogdir=/users/apple/zookeeper/logs
clientport=4180
Parameter description:
The base time unit, millisecond value, used in Ticktime:zookeeper.
DataDir: Data directory. can be any directory.
The Datalogdir:log directory can also be any directory. If this parameter is not set, the same settings are used and DataDir.
ClientPort: Listen for the port number of the client connection
4.2 Pseudo-Cluster mode
The so-called pseudo-cluster, is to start multiple zookeeper processes in a single machine, and form a cluster. Take the example of starting 3 zookeeper processes.
Copy 2 copies of the Zookeeper directory:
The Zookeeper0/conf/zoo.cfg file is:
ticktime=2000
Initlimit=5
synclimit=2
Datadir=/users/apple/zookeeper0/data
Datalogdir=/users/apple/zookeeper0/logs
clientport=4180
server.0=127.0.0.1:8880:7770
server.1=127.0.0.1:8881:7771
server.2=127.0.0.1:8882:7772
Several new parameters have been added, meaning the following:
1 The Initlimit:zookeeper cluster contains multiple servers, one of which is leader, and the rest of the servers in the cluster are follower. The Initlimit parameter configures the maximum heartbeat time between follower and leader when the connection is initialized. At this point the parameter is set to 5, stating that the time limit is 5 times times ticktime, or 5*2000=10000ms=10s.
2 Synclimit: This parameter configures the maximum length of time to send messages, requests, and responses between leader and follower. At this point the parameter is set to 2, stating that the time limit is twice times ticktime, or 4000ms.
3 server. X=A:B:C where x is a number that indicates which is the first server. A is the IP address where the server resides. b Configure the port used by the server and the leader in the cluster to exchange messages. C Configure the port to use when electing leader. Because the pseudo-cluster mode is configured, the B and C parameters of each server must be different.
Refer to Zookeeper0/conf/zoo.cfg, configure Zookeeper1/conf/zoo.cfg, and zookeeper2/conf/zoo.cfg files. Just change the DataDir, Datalogdir, clientport parameters.

Create a new myID file in the previously set DataDir, write a number that indicates the number of server. The number must correspond to x one by one in the server.x in the Zoo.cfg file.
Write 0 in the/users/apple/zookeeper0/data/myid file, write 1 in/users/apple/zookeeper1/data/myid file,/users/apple/zookeeper2/data/ Write 2 in the myID file.

Enter/users/apple/zookeeper0/bin,/users/apple/zookeeper1/bin,/users/apple/zookeeper2/bin three directories, and start the server respectively.
4.3 Cluster mode
Cluster mode configuration and pseudo-cluster are basically consistent.
Because the server is deployed on different machines in cluster mode, the CONF/ZOO.CFG files for each server can be identical.
Here is an example:
ticktime=2000
Initlimit=5
synclimit=2
Datadir=/home/zookeeper/data
Datalogdir=/home/zookeeper/logs
clientport=4180
server.43=10.1.39.43:2888:3888
server.47=10.1.39.47:2888:3888
server.48=10.1.39.48:2888:3888
The example deploys 3 zookeeper servers, each deployed on 10.1.39.43, 10.1.39.47, 10.1.39.48. It is important to note that the numbers in the myID file in the DataDir directory of each server must be different, 10.1.39.43 server myID is, 10.1.39.47 server's myID is 47, 10.1.39.48 The server's myID is 48.
5 Zookeeper Common commands
Zookeeper Service Command:
1. Start ZK service:./zkserver.sh start
2. View ZK Service status:./ZKSERVER.SH status
3. Stop ZK Service:./zkserver.sh stop
4. Restart the ZK service:./zkserver.sh restart
ZK Client Command:
The ZooKeeper command-line tool is similar to the Linux shell environment, which allows access to ZooKeeper, data creation, data modification, and so on. Using Zkcli.sh-server 127.0.0.1:2181 to connect to the ZooKeeper service, the system outputs the relevant environment and configuration information for ZooKeeper when the connection is successful.
Some simple operations of the command-line tool are as follows:
1. Display the root directory, file: LS/use the LS command to view the content contained in the current ZooKeeper
2. Display the root directory, file: LS2/View the current node data and can see the number of updates and other data
3. Create the file and set the initial content: Create/zk "Test" creates a new Znode node "ZK" and the string associated with it
4. Get the contents of the file: Get/zk confirm if Znode contains the string we created
5. Modify the contents of the file: Set/zk "Zkbak" sets the string associated with ZK
6. Delete file: Delete/zk The znode you just created is deleted
7. Exit the client: Quit
8. Help command:
ZooKeeper commonly used four word command:
ZooKeeper supports the interaction of some specific four-word command letters. Most of them are query commands that are used to obtain the current state of the ZooKeeper service and related information. The user can submit the appropriate command to the ZooKeeper via Telnet or NC at the client
1. You can use the command: Echo stat|nc 127.0.0.1 2181来 to see which node is selected as follower or leader
2. Use echo ruok|nc 127.0.0.1 2181 to test if the server is started, if the reply Imok indicates that it has been started.
3. Echo dump| NC 127.0.0.1 2181, which lists the unprocessed sessions and temporary nodes.
4. Echo Kill | NC 127.0.0.1 2181, turn off server
5. Echo conf | NC 127.0.0.1 2181, output the details of the related service configuration.
6. Echo Cons | NC 127.0.0.1 2181, which lists full connection/session details for all clients connected to the server.
7. Echo envi |nc 127.0.0.1 2181, output detailed information about the service environment (as distinct from the Conf command).
8. Echo reqs | NC 127.0.0.1 2181, which lists the unprocessed requests.
9. Echo Wchs | NC 127.0.0.1 2181, listing the details of the server watch.
echo WCHC | NC 127.0.0.1 2181, the session lists the details of the server watch, and its output is a list of watch-related sessions.
echo WCHP | NC 127.0.0.1 2181, which lists the details of the server watch by path. It outputs a path associated with the session.

Introduction to Zookeeper working principle, installation configuration, tool commands