Build a highly Available zookeeper cluster (reprint)

ZooKeeper is a top-of-the-range Apache project that provides distributed infrastructure services such as data publishing/subscription, load Balancing, naming services, distributed coordination/notification, and distributed locks, providing efficient, high-availability distributed coordination services for distribution applications. Thanks to its convenient usage, excellent performance and good stability, ZooKeeper is widely used in large distributed systems such as Hadoop, HBase, Kafka and Dubbo.

The target reader of this paper is to have a certain understanding of ZooKeeper technical staff, will be from the ZooKeeper operation mode, cluster composition, disaster tolerance and horizontal expansion four aspects of the gradual deepening, and finally build a highly available ZooKeeper cluster.

Operating mode

The Zookeeper has three modes of operation: stand-alone mode, pseudo-cluster mode and cluster mode.

Stand-alone mode

This model is generally applicable to the development of testing environment, on the one hand we do not have so many machine resources, in addition to the usual development and debugging does not require excellent stability.

Running stand-alone mode in a Linux environment requires the following steps:

　　1. Preparing the Java Runtime environment

Install Java 1.6 or later JDK, and configure Java-related environment variables $JAVA _home.

　　2. Download the ZooKeeper installation package

: http://zookeeper.apache.org/releases.html. Select the latest stable version and extract to the specified directory, we use the $ZK _home to represent the directory.

　　3. Configure Zoo.cfg

The first time you use ZooKeeper, you need to rename the Zoo_sample.cfg file under $ZK _home to zoo.cfg and configure the following

ticktime=2000    # #Zookeeper最小时间单元, Unit milliseconds (ms), default is 3000datadir=/var/lib/zookeeper    # #Zookeeper服务器存储快照文件的目录, Must be configured
Datalogdir=/var/lib/log     # #Zookeeper服务器存储事务日志的目录, the default is datadirclientport=2181    # #服务器对外服务端口, Typically set to 2181initlimit=5    # #Leader服务器等待Follower启动并完成数据同步的时间, the default value is 10, which represents 10 times times the Ticktime synclimit=2    # # The maximum delay time for heartbeat detection between the leader server and follower, with a default value of 5 representing 5 times times the Ticktime

　　4. Start the service

Start the service using the zkserver.sh script in the $ZK _home/bin directory.

Cluster mode

A ZooKeeper cluster usually consists of a set of machines, typically more than 3 units can be formed into a usable ZooKeeper cluster.

Each machine that makes up the ZooKeeper cluster maintains the current server state in memory, and each machine maintains communication with one another. The important point is that as long as more than half of the machines in the cluster work properly, the entire cluster will be able to serve as normal.

The ZooKeeper client program chooses to create a TCP connection with any server in the cluster, and once the client and server are disconnected, the client automatically connects to the other servers in the cluster.

So how do you run ZooKeeper cluster mode? First, if we have three servers, the IP is IP1, IP2, and IP3, then we need to perform the following steps:

　　1. Preparing the Java Runtime Environment (IBID.)

　　2. Download the ZooKeeper installation package (IBID.)

　　3. Configure Zoo.cfg

Ticktime=2000datadir=/var/lib/zookeeperdatalogdir=/var/lib/logclientport=2181initlimit=5synclimit=2server.1=ip

1

: 2888:3888server.2=ip

2

: 2888:3888server.3=ip

3

: 2,888:3,888

As you can see, the cluster mode has more SERVER.ID=HOST:PORT1:PORT2 configuration than the standalone mode. Where the ID is referred to as the server ID, which identifies the machine serial number in the cluster (the myID file is created in the DataDir directory of each machine, and the file content is the server ID number for that machine). The host machine Ip,port1 is used to specify the ports that the Follower server communicates with the Leader server for communication and data synchronization, and Port2 is used for voting communication during the Leader election process.

　　4. Create a myID file

Create a file named myID in the DataDir directory, and write the corresponding Server ID on the first line of the file.

　　5. Follow the same steps to configure ZOO.CFG and myID files for other machines

　　6. Start the service

Pseudo-Cluster mode

This is a special cluster mode in which all the servers in the cluster are deployed on a single machine. When you have a better machine on hand, if deployed as a stand-alone mode, you waste resources, in which case, ZooKeeper allows you to launch multiple ZooKeeper service instances on a single machine by starting different ports to serve as a cluster feature.

In this mode, you only need to change the zoo.cfg as follows:

Ticktime=2000datadir=/var/lib/zookeeperdatalogdir=/var/lib/logclientport=2181initlimit=5synclimit=2server.1=ip

1

: 2888:3888server.2=ip

1

: 2889:3889server.3=ip

1

: 2,890:3,890

Cluster composition

To build a highly available ZooKeeper cluster, we first need to determine the size of the cluster. About the ZooKeeper Cluster Server composition, believe that many of the ZooKeeper understand but not enough depth of the reader, there are or have ever existed such a mistaken understanding: in order to make the ZooKeeper cluster can successfully elect Leader, must be The number of servers in the ZooKeeper cluster is deployed as odd . One thing we need to clarify here is that any ZooKeeper server can be deployed and functioning properly.

So what is this mistaken understanding that exists in so many readers? In fact, about the number of ZooKeeper cluster servers, ZooKeeper official did give a proposal on odd numbers, but most of the ZooKeeper users of this advice to understand the deviation. In the " Half Alive " feature mentioned earlier in this book, we have learned that a ZooKeeper cluster must have more than half of the machines in the cluster to work properly and communicate with each other properly if it is to provide available services externally. Based on this feature, if you want to set up a cluster that allows N machines to fall down, deploy a ZooKeeper cluster of 2*n+1 servers. As a result, a ZooKeeper cluster of 3 machines can still work after 1 machines have been hung up, and for a ZooKeeper cluster of 5 servers, it can be disaster-tolerant for 2 machines to hang out. Note that if you are a ZooKeeper cluster of 6 servers, you can only hang up 2 machines, because if you hang up 3, the remaining machines will not be able to achieve more than half.

Therefore, from the above explanation, we can actually see, for a 6 machine composed of ZooKeeper cluster, and a 5 machines composed of ZooKeeper cluster, its capacity for disaster tolerance does not have any significant advantages, but more occupy a server resources. For this reason, ZooKeeper clusters are typically designed to deploy generate several servers.

Disaster tolerance

The so-called disaster recovery, in the IT industry, usually refers to the ability of our computer information systems to provide available services externally when subjected to devastating disasters such as fire, earthquakes, power outages, and other basic network equipment failures.

For some common applications, in order to meet the disaster tolerance standard, we usually choose to deploy on multiple machines to form a cluster, so that even if one or several machines in the cluster fail, the entire cluster will still be able to provide services available externally.

And for some core applications, not only through the use of multiple machines to build a cluster to provide services, but also to deploy the machine in the cluster in two rooms, so that even if one of the rooms encounter a disaster, can still provide external services available.

It's all about the application-level disaster-tolerant model, so how can disaster recovery be done for the underlying component of ZooKeeper? In this case, perhaps a number of readers will have doubts, ZooKeeper since has solved the single point problem, then why do disaster tolerance?

Single point problem

Single point problem is one of the most common and classic problems in distributed environment, and it is a single point problem in many distributed systems. Specifically, the single point problem is that in a distributed system, if a component fails, it can cause the whole system's usability to be greatly reduced or even paralyzed, then we think that the component has a single point of problem.

The ZooKeeper has really solved the single point problem very well. We have learned that, based on the "more than half" design principle, ZooKeeper has at least half of the machines in the cluster keep up-to-date data during operation. As a result, as long as more than half of the machines in the cluster work properly, the entire cluster can be serviced externally.

Disaster tolerance

Solve the problem of a single point, is it time to consider disaster? The answer is no, there is still a need to consider disaster tolerance when building a high-availability cluster. As mentioned above, if more than half of the machines in the cluster are still working properly, the cluster will be able to provide normal service to the outside. Then, if the entire computer room has a catastrophic accident, then obviously is not a single point of the category of problems.

During the design of the disaster tolerant scheme of ZooKeeper, we should fully consider the "half principle". In other words, no matter what happens, we have to make sure that more than half of the machines in the ZooKeeper cluster work properly. Therefore, the following two deployment scenarios are usually available.

Dual Room Deployment

When designing a disaster-tolerant solution, we usually consider the problem in the room. In reality, many of the company's computer room size is not large, so the two-room deployment is a more common scenario. But unfortunately, in the current version of the ZooKeeper, there is no way to achieve a better disaster tolerance in the double room conditions-because no matter what happens in the engine room, it is possible to make the ZooKeeper cluster can not more than half of the machines available. Of course, in a scene with two rooms, there is usually a room is the main engine room (in general, the company will spend more money to rent a more stable, more reliable equipment room, this room is the main engine room, and another room is more inexpensive). The only thing we can do is to try to deploy more machines in the main engine room. For example, for a ZooKeeper cluster of 7 machines, 4 machines are usually deployed in the main room, and the remaining 3 machines are deployed to another room.

Three Machine room deployment

Since in the two-room deployment mode and can not achieve good disaster tolerance, then for the conditional company, choose three room deployment is undoubtedly a better choice, no matter which engine room has failed, the remaining two machine rooms more than half the number of machines. If we have three rooms can be deployed services, and the network in the three rooms are in good condition, then you can deploy a number of machines in three rooms to form a ZooKeeper cluster.

We assume that the total number of machines that comprise the ZooKeeper cluster is N, the number of ZooKeeper servers deployed in three engine rooms are N1, N2, and N3 respectively, and if the ZooKeeper cluster is to have good disaster tolerance, we can calculate the ZooKeeper cluster based on the following algorithm. Machine deployment Scenarios.

1. Calculate N1

If the total number of servers for the ZooKeeper cluster is N, then:

N1 = (N-1)/2

In Java, the "/" operator automatically takes the result of the calculation down to its full operation. For example, if n=8, then n1=3, if n=7, then N1 is equal to 3.

2. Calculate optional values for N2

N2 's calculation rules are very similar to N1, except that the value of N2 is within a range of values:

The value range of N2 is 1~ (N-N1)/2

That is, if n=8, then n1=3, then the value range of N2 is one or two, respectively, 1 and 2. Note that 1 and 2 are only N2 optional values, not final values-if N2 is an optional value, the value of N3 cannot be computed, and the optional value is not valid.

3. Calculate N3, while determining the value of N2

Obviously, now only N3, you can simply think that the value of N3 is the remaining number of machines, namely:

N3 = n-n1-n2

Only the N3 value must meet N3 < n1+n2. On the basis of satisfying this condition, we traverse the optional value of the N2 calculated in step 2, we can get the number of servers in each room when the three machine rooms are deployed.

Now we take 7 machines as an example to see how to allocate the machine distribution of the three machine rooms. Based on Step 1 of the algorithm, we first determine that the value of N1 is 3. Based on step 2 of the algorithm, we determined that the optional values for N2 are 1 and 2. Finally, based on step 3, we traverse the optional values of the N2 to get two deployment scenarios, namely (3,1,3) and (3,2,2). The following is a simple implementation of the Java program code for the above algorithm:

public class Allocation {        static final int n = 7;    public static void Main (string[] args) {        int n1,n2,n3;        N1 = (n-1)/2;        int N2_max = (n-n1)/2;        for (int i=1; i<=n2_max; i++) {            n2 = i;            n3 = n-n1-n2;            if (N3 >= (n1+n2)) {                continue;            }            System.out.println ("(" +n1+ "," +n2+ "," +n3+ ")");}}}    

Horizontal expansion

Horizontal scalability can be said to be a distributed system in the high availability of the basic, is also a very important requirement, through the level of expansion can help the system without or do little improvement work, the premise of rapid improvement of the system external service support capacity. Simply put, horizontal expansion is to add more machines to the cluster to improve the service quality of the system.

Unfortunately, the ZooKeeper is not perfect for expanding horizontally, and requires a reboot of the entire cluster. There are usually two ways to reboot, one is to restart the cluster as a whole, and the other is to restart the server on a per-station basis.

Overall restart

The so-called cluster reboot is to stop the whole cluster first, then update the ZooKeeper configuration and start again. If ZooKeeper is not a very core component in your system and can allow a short service stop (usually a few seconds), you might choose this way. During the overall restart, all clients of the cluster cannot connect to the cluster. By the time the cluster starts again, these clients will be able to connect automatically-note that the client session that was established before the overall startup is not invalidated by this overall restart. That is, the time spent during the overall restart is not counted in the calculation of the session timeout period.

Reboot by console

This approach is more suitable for most real-world scenarios. In this way, only one machine in the cluster is restarted at a time, and the machines in the entire cluster are restarted on a per-station basis. This method can still guarantee the normal service of the cluster during the restart.

Original address: http://www.cnblogs.com/cyfonly/p/5626532.html

Build a highly Available zookeeper cluster (reprint)