[Hadoop] [Fundamentals]zookeeper Fundamentals

1. Introduction

https://www.ibm.com/developerworks/cn/opensource/os-cn-zookeeper/

2. Data Model

Zookeeper maintains a hierarchical relational data structure that is very similar to a standard file system:

　　　　　　　　

Zookeeper This data structure has the following features:

1. Each subdirectory entry, such as Nameservice, is called Znode, and the Znode is uniquely identified by the path it is located in, such as Server1, which is the Znode identity/nameservice/server1
2. Znode can have child node directories, and each znode can store data, note that ephemeral types of directory nodes cannot have child node directories
3. Znode is a version of the data stored in each Znode can have multiple versions, that is, one access path can store multiple copies of data
4. Znode can be a temporary node, once created this Znode client and the server lost contact, this znode will also be automatically deleted, Zookeeper client and server communication with a long connection, each client and server through the heartbeat to maintain connectivity, this connection status is called Session, if Znode is a temporary node, this session expires, Znode also deleted
5. Znode directory name can be automatically numbered, such as App1 already exists, and then created, will be automatically named as App2
6. Znode can be monitored, including changes in the data stored in this directory node, changes in sub-node directories, and so on, once the change can notify the settings monitoring client, this is the core feature of Zookeeper, Zookeeper Many of the functions are based on this feature is implemented, There will be examples in the following typical application scenarios

Typical application scenarios for ZooKeeper

Zookeeper from a design pattern perspective, it is a distributed service management framework based on the observer pattern design that stores and manages the data that everyone cares about and then accepts the viewer's registration, and once the status of the data changes, Zookeeper will be responsible for notifying Zookeeper registered on the observers to make corresponding response, so as to achieve similar Master/slave management mode in the cluster, about the detailed architecture of Zookeeper and other internal details can read Zookeeper source code

Here's a detailed description of these typical scenarios, which is how Zookeeper can help us solve those problems? The answer is given below.

Unified Naming Services (name service)

In distributed applications, a complete set of naming conventions is often required, both to produce a unique name and to be easily recognizable and remembered, usually with a tree-shaped name structure as an ideal choice, and the tree-shaped name structure is a hierarchical directory structure that is neither friendly nor repetitive. Speaking of which, you might think of Jndi, yes. Zookeeper's name service is similar to what JNDI can do, and they all associate hierarchical directory structures to certain resources, but Zookeeper's name service is more broadly Association, maybe you don't need to associate a name to a specific resource, you might just need a name that doesn't duplicate, just like a unique numeric primary key in the database.

The Name Service is already a built-in feature of Zookeeper, and you can do so just by invoking the Zookeeper API. It is easy to create a directory node if you call the Create interface.

Configuration management (config Management)

Configuration management is common in distributed application environments, where multiple PC servers are required for the same application system, but some of the configuration items of the applications they run are the same, and if you want to modify these same configuration items, you must modify each PC server that is running the application. This is very troublesome and error prone.

Configuration information like this can be left to Zookeeper to manage, the configuration information is stored in a directory node in Zookeeper, and then all the application machine needs to be modified to monitor the configuration information status, once the configuration information changes, each application machine will receive Zookeeper notification , and then get new configuration information from Zookeeper to the system.

Figure 2. Configuration management structure diagram

Cluster Management (Group membership)

Zookeeper can easily realize the function of cluster management, if there are more than one server to form a service cluster, then a "manager" must know the service status of each machine in the current cluster, and once the machine cannot provide the service, other clusters in the cluster must know to make the adjustment and redistribution service policy. Also, when you increase the service capability of a cluster, you add one or more servers, and you must also let "Explorer" know.

Zookeeper not only helps you maintain the service status of the machines in your current cluster, but also helps you select a "manager" to manage the cluster, which is another function of Zookeeper Leader election.

They are implemented by creating a directory node of type ephemeral on Zookeeper, and then each Server invokes the GetChildren (String path, Boolean Watch) method on the parent directory node on which they create the directory node and sets Watch is true, because it is the ephemeral directory node, when the Server that created it dies, the directory node is deleted, so children will change, and Watch on GetChildren will be called, so other servers will know There is already a server dead. New Server is the same principle.

Zookeeper How to implement Leader election, which is to choose a Master Server. As in the previous one, each server creates a ephemeral directory node, but it is also a sequential directory node, so it is a ephemeral_sequential directory node. The reason it is a ephemeral_sequential directory node is because we can give each server number, we can choose the server that is currently the smallest number is the Master, if this smallest number of the server died, because it is the ephemeral node, The node for the dead Server is also deleted, so a node with the smallest number appears in the current node list, and we select the node as the current Master. In this way, the dynamic selection master is realized, which avoids the problem of single-point failure in the traditional sense of single-master.

Figure 3. Cluster management structure diagram

The sample code for this section is as follows, and the complete code is shown in the attachment:

Listing 3. Leader election key code

void Findleader () throws Interruptedexception {         byte[] leader = null;         try {             leader = Zk.getdata (root + "/leader", true, null),         } catch (Exception e) {             logger.error (e);         }         if (leader! = NULL) {             following ();         } else {             String newleader = null;             try {                 byte[] localhost = inetaddress.getlocalhost (). getaddress ();                 Newleader = zk.create (root + "/leader", localhost,                 ZooDefs.Ids.OPEN_ACL_UNSAFE, createmode.ephemeral);             } catch (Exception e) {                 logger.error (e);             }             if (Newleader! = null) {                 leading ();             } else {                 mutex.wait ();     }}}

Shared Lock (Locks)

Shared locks are easily implemented in the same process, but not across processes or between different servers. Zookeeper it is easy to implement this function, the implementation is also required to obtain a lock Server to create a ephemeral_sequential directory node, and then call GetChildren method gets the smallest directory node in the current directory node list is not the directory node that it created itself, if it is created by itself, then it acquires this lock, if not then it calls exists (String path, Boolean watch) Method and monitor the changes in the list of directory nodes on Zookeeper, until the node that you create is the smallest directory node in the list, so that the lock is easy to release, as long as you delete the directory node that you created earlier.

Figure 4. Zookeeper implementation of Locks flowchart

The implementation code of the synchronization lock is as follows: The complete code is shown in the attachment.

Listing 4. Key code for Sync lock

void Getlock () throws Keeperexception, interruptedexception{         list<string> List = Zk.getchildren (root, False) ;         string[] nodes = List.toarray (new string[list.size ());         Arrays.sort (nodes);         if (Myznode.equals (root+ "/" +nodes[0])) {             doAction ();         }         else{             Waitforlock (nodes[0]);         }     }     void Waitforlock (String lower) throws Interruptedexception, keeperexception {        stat stat = zk.exists (root + "/" + Lowe R,true);         if (stat! = null) {             mutex.wait ();         }         else{             Getlock ();         }     }

Queue Management

Zookeeper can handle two types of queues:

1. This queue is available when a member of a queue is NAND, otherwise it waits for all members to arrive, which is the synchronization queue.
2. Queues are queued and out-of-line in a FIFO manner, for example to implement producer and consumer models.

The implementation of the synchronization queue with Zookeeper is as follows:

Create a parent directory/synchronizing, each member of the monitor flag (Set Watch) bit directory/synchronizing/start exists, and then each member joins the queue, the way to join the queue is to create/synchronizing/ Member_i The temporary directory node, and then each member gets/synchronizing all directory nodes of the directory, that is, Member_i. Determine if the value of I is already the number of members, and if it is less than the number of members waiting for/synchronizing/start to appear, create/synchronizing/start if it is already equal.

It is easier to understand with the following flowchart:

Figure 5. Synchronization Queue Flowchart

The key code for the synchronization queue is as follows, and the complete code is shown in the attachment:

Listing 5. Synchronization queue

void Addqueue () throws Keeperexception, interruptedexception{         zk.exists (root + "/start", true);         Zk.create (root + "/" + Name, new byte[0], Ids.open_acl_unsafe,         createmode.ephemeral_sequential);         Synchronized (mutex) {             list<string> List = Zk.getchildren (root, false);             if (List.size () < size) {                 mutex.wait ();             } else {                 zk.create (root + "/start", New byte[0], Ids.open_acl_unsa FE,                 createmode.persistent);}}}  

When the queue is not full to wait () and then waits for watch's notification, watch's code is as follows:

public void process (Watchedevent event) {         if (Event.getpath (). Equals (Root + "/start") &&         Event.gettype () = = Event.EventType.NodeCreated) {             System.out.println ("Get Notified");             Super.process (event);             DoAction ();         }     }

FIFO queue with Zookeeper implementation ideas are as follows:

The idea of implementation is also very simple, that is, in a specific directory to create a sequential type subdirectory/queue_i, so that all members can be added to the queue is numbered, out of the queue through the GetChildren () method can return all the elements of the current queue, Then consume the smallest one, so that the FIFO can be guaranteed.

Below is a sample code for the form of a queue of producers and consumers, the complete code can be seen in the attachment:

Listing 6. Producer Code

Boolean Produce (int i) throws Keeperexception, interruptedexception{         bytebuffer b = bytebuffer.allocate (4);         Byte[] value;         B.putint (i);         Value = B.array ();         Zk.create (root + "/element", Value, ZooDefs.Ids.OPEN_ACL_UNSAFE,                     createmode.persistent_sequential);         return true;     }

Listing 7. Consumer Code

int consume () throws Keeperexception, interruptedexception{         int retvalue =-1;         Stat stat = null;         while (true) {             synchronized (mutex) {                 list<string> List = Zk.getchildren (root, true);                 if (list.size () = = 0) {                     mutex.wait ();                 } else {                     integer min = new Integer (list.get (0). SUBSTRING (7));                     for (String s:list) {                         Integer tempvalue = new Integer (s.substring (7));                         if (Tempvalue < min) min = tempvalue;                     }                     Byte[] B = zk.getdata (root + "/element" + min,false, stat);                     Zk.delete (root + "/element" + min, 0);                     Bytebuffer buffer = Bytebuffer.wrap (b);                     RetValue = Buffer.getint ();                     return RetValue;}}}  

Back to top of page

Summarize

Zookeeper, a sub-project in the Hadoop project, is an essential module for Hadoop cluster Management, which is used primarily to control the data in the cluster, such as its management of NameNode in Hadoop clusters, and the Master election in Hbase, S State synchronization between Erver and so on.

This article describes the basics of Zookeeper and describes several typical application scenarios. These are the basic functions of Zookeeper, the most important is that Zoopkeeper provides a good set of distributed cluster management mechanism, it is the hierarchical type of directory tree data structure, and the tree nodes in the effective management, so that can design a variety of distributed data management model, Rather than just a few of the common scenarios mentioned above.

[Hadoop] [Fundamentals]zookeeper Fundamentals