Zookeeper Dynamic Update server list

--------------------------------------------------------------------------------------

[Copyright: The author of this article is original, reproduced please indicate the source]

Article Source: http://www.cnblogs.com/sdksdk0/p/5585192.html
Juppé Id:sdksdk0
-----------------------------------------------------------------------------------

What we are sharing today is the zookeeper of big data practices.

Zookeeper internal is a cluster, the main node is elected, the external looks like only one, the preservation of a state data. When doing distributed application coordination, it can reduce the development difficulty.

With high availability, loosely coupled interaction mode.

The configuration of zookeeper can be viewed in my article: http://blog.csdn.net/sdksdk0/article/details/51517460, where the installation configuration is not repeated.

One, Zookeeper API interface

Zookeeper in general, the data stored in no more than 1M. Mainly to save some configuration information, the main feature is the real-time monitoring data update.


Second, the main application 1, cluster Management: The minimum number of provisions for the master, so when we monitor the Servers node, we get a list of servers, as long as all the cluster machine logic that the minimum number node is master, then Master is selected, When this master goes down, the corresponding znode disappears, and then the new server list is pushed to the client, then each node logic thinks the minimum number node is master, so that the dynamic master election is done.


2, Configuration Management: In Distributed application environment is very common, for example, the same application system requires more than one PC server to run, but they run some of the application system configuration items are the same, if you want to modify these same configuration items, then you must modify each PC running this application system server, This is very troublesome and error prone. Save the configuration information in a directory node in Zookeeper, and then monitor the status of configuration information for all application machines that need to be modified, and once the configuration information changes, each application machine receives a ZOOKEEPER notification and then gets the new configuration information from the Zookeeper to be applied to the system.


3. Shared Lock: It is easy to implement in the same process, but not in cross-process or between different servers. Zookeeper it is easy to implement this function, the implementation is also required to obtain a lock Server to create a ephemeral_sequential directory node, and then call GetChildren method gets the smallest directory node in the current directory node list is not the directory node that it created itself, if it is created by itself, then it acquires this lock, if not then it calls exists (String path, Boolean watch) Method and monitor the changes in the list of directory nodes on Zookeeper, until the node that you create is the smallest directory node in the list, so that the lock is easy to release, as long as you delete the directory node that you created earlier.


4. Queue management: Zookeeper can handle two types of queues: This queue is available when a member of a queue is NAND, otherwise it waits for all members to arrive, this is a synchronous queue, and queues are queued and out-of-line in a FIFO manner, such as implementing producer and consumer models


Third, using Eclipse Connection zookeeper in eclipse, we can import the required packages, and then add and remove the nodes to check operations. This can be done when connecting, where I use 3 zookeeper to operate. ubuntu1,2,3 are host names, respectively.

ZooKeeper ZK = null; @Beforepublic void init () throws exception{    ZK = new ZooKeeper ("ubuntu2:2181,ubuntu1:2181, Ubuntu3:2181 ", Watcher, New () {        //] callback method when the listener event occurs        @Override public        void Process (Watchedevent event) {            System.out.println (Event.getpath ());            System.out.println (Event.gettype ());            }    );}

Create a node: I'm here to create a permanent node with a temporary node and a permanent node in the Zookeeper node. To create an Eclipse node with the directory, the encoding format of the content is utf-8,ids refers to the permission control, which I use is the open ACL permission control. Finally, the stream needs to be closed.

@Testpublic void Testzknode () throws Exception {    String path = zk.create ("/eclipse", "command sink Technology". GetBytes ("Utf-8"), Ids.open_acl_unsafe, createmode.persistent);    System.out.println ("created a permanent node:" + path);    Zk.close ();}

Then there is the registration listener, after all zookeeper has a very important function is to listen to the status of the entire service.

@Testpublic void Testget () throws Exception {    //listener registration can only take effect once    byte[] data = Zk.getdata ("/eclipse", True, new Stat ( ));    System.out.println (new String (data, "utf-8"));    Thread.Sleep (Long.max_value);}

The execution is called in the Main method.

@Testpublic void Testset () throws Unsupportedencodingexception, Keeperexception, interruptedexception{    Zk.setdata ("/eclipse", "Who is the Hero"). GetBytes ("Utf-8"), 1);    Zk.close ();}

Four, dynamic server because I mainly share is how on the client dynamic listening server on-line and offline, so we first to write a server process.


First of all, we need to define the node information that we need later, first go to the zookeeper client to run it, create a Grpnode node for our follow-up operation.

Private ZooKeeper zk;private String groupnode = "Grpnode";p rivate String subnode = "Sub";//Register information to ZooKeeper public void conn Ectzk (String name) throws Keeperexception, Interruptedexception, IOException {    ZK = new ZooKeeper ("ubuntu2:2181, Ubuntu1:2181,ubuntu3:2181 ", Watcher, New () {                //) callback method when the listener event occurs                @Override public                void Process ( Watchedevent event) {                }            });    String Path = zk.create ("/" + Groupnode + "/" + subnode, Name.getbytes (), Ids.open_acl_unsafe, Createmode.ephemeral_sequen TIAL);    SYSTEM.OUT.PRINTLN ("Server on line, created a child node:" + path);}

The next step is to zookeeper the default business logic and finally call it in the Main method. Of course, you can put this into a jar package and put it on Hadoop to run.

Business processing logic public void handle () throws Exception {    thread.sleep (long.max_value);} public static void Main (string[] args) throws Exception {    if (args.length==0) {        System.err.println ("The number of arguments is incorrect, Please attach the server name as a parameter to start ...);        System.exit (1);    }    Go to Zookeeper Register this server information    AppServer server = new AppServer ();    Server.connectzk (Args[0]);    Server.handle ();}

Five, dynamic client server after writing, we need a client to listen to the server's upper and lower line operation. A zookeeper listener callback method is also used. Once the server has changed, you can hear it dynamically. The main is to monitor the change of parent-child nodes.

Private  volatile   list<string>  servers;private ZooKeeper zk;//action to trigger server update with ZK listener function public void Connectzk () throws IOException, Keeperexception, interruptedexception{    ZK = new ZooKeeper ("UBUNTU2:2181,UBUNTU1 : 2181,ubuntu3:2181 "," Watcher "," new "() {                //) callback method when the listener event occurs                @Override public                void Process (watchedevent Event) {                    if ("/grpnode". Equals (Event.getpath ()) && Event.gettype () ==eventtype.nodechildrenchanged) {                    //Trigger Update server list action                    try {                        updateserverlist ();                    } catch (Exception e) {                        e.printstacktrace ();                    }                     }                }            });            Updateserverlist ();        }

Dynamic Fetch Server list, here is mainly to monitor the change of parent-child node.

Dynamic Fetch server list public void Updateserverlist () throws Keeperexception, Interruptedexception, unsupportedencodingexception {    arraylist<string>  serverlist=new arraylist<string> ();    Listens for child nodes, and registers listeners with the parent node    list<string>  childer=zk.getchildren ("/grpnode", true);    Traverse child node    for (String child:childer) {        byte[] data=zk.getdata ("/grpnode/" +child,false, New Stat ());        String Server=new string (data, "Utf-8");        Save the retrieved server name to list        serverlist.add (server);    }    Put the staged list in the global list    servers=serverlist;    System.out.println ("The latest online server is:" +serverlist);}

And finally the main method that we're most familiar with.

Client's business function public void handle () throws interruptedexception{    Thread.Sleep (long.max_value);} public static void Main (string[] args) throws IOException, Interruptedexception, keeperexception{    appclient client= New Appclient ();    Client.connectzk ();    Client.handle ();}

Running the results in Hadoop is as follows:

Summary: Zookeeper as a subproject in the Hadoop project is an essential module for Hadoop cluster Management, which is used primarily to control the data in the cluster, such as its management of NameNode in Hadoop clusters, and the Master electio in Hbase N, state synchronization between servers, and so on. Zoopkeeper provides a good set of distributed cluster management mechanism, it is the hierarchical type of directory tree data structure, and the tree nodes in the effective management, so that can design a variety of distributed data management model.

Zookeeper Dynamic Update server list