Zookeeper distributed lock and queue Implementation Example tutorial

In the distributed system, often need some of the distributed synchronization primitives to do some work together, the last article introduced the basic principle of zookeeper, this article describes the implementation of lock and queue based on zookeeper, The main code comes from Zookeeper's official recipe. The

lock

Fully distributed lock is globally synchronized, which means that no two clients at any given time will assume that they all have the same locks, and that zookeeper can be used to implement distributed locks. You need to first define a lock node (lock root). The client that needs to acquire the lock uses the following steps to obtain the lock:

    Guarantee the presence of the parent root node of the lock node (lock root), which is shared by each lock client to be acquired. This node is persistent.

    The first time you need to create a node for which this client will get lock, call Create (), and set the node to the Ephemeral_sequential type, which indicates that the node is temporary and sequential.

    calls GetChildren () on the parent lock node, which does not require a monitoring flag to be set. (In order to avoid "herding effect").

    In accordance with the principle of fair competition, the child nodes in step 3 (the node to acquire the lock) are sorted according to the order of the nodes, the smallest node is taken as lock owner, and the node ID
    whether is the owner ID, if it is returned, lock succeeded. If not, call exists () to listen for the ID of the previous bit smaller than yourself, focusing on its lock-releasing operation (i.e., exist watch).

    If the watch of step 4th listening exist is triggered, continue to determine if you can get lock by the principle in 4.

to release the lock: the client that needs to release the lock needs to delete only the node created in step 2nd.

Note:

A deletion of a node can only cause a client to be awakened because each node is watch by only one client, which avoids the "herding effect".

Implementation of a distributed lock:

/**  *  * Licensed to the Apache Software Foundation  (ASF)   Under one or more  * contributor license agreements.  see the  notice file distributed with  * this work for additional 
Information regarding copyright ownership.  * the asf licenses this file to you under the apache  license, version 2.0  *  (the  "License");  you may not use  this file except in compliance with  * the License.   You may obtain a copy of the license at  *  * http:// www.apache.org/licenses/LICENSE-2.0  *  * unless required by applicable law  or agreed to in writing, software  * distributed under the License is distributed on an  "As is"  basis,  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, 
Either express or implied.  * See the License for the specific language governing 
Permissions and  * limitations under the license.
 */package org.apache.zookeeper.recipes.lock;
import org.slf4j.logger;
import org.slf4j.loggerfactory;
import org.apache.zookeeper.keeperexception;
import org.apache.zookeeper.watchedevent;
import org.apache.zookeeper.watcher;
import static org.apache.zookeeper.createmode.ephemeral_sequential;
import org.apache.zookeeper.zookeeper;
import org.apache.zookeeper.data.acl;
import org.apache.zookeeper.data.stat;
import java.util.list;
import java.util.sortedset; import java.util.trEeset; /**  * a <a href= "package.html" >protocol to implement an  Exclusive  *  write lock or to elect a leader</a>. < p/> you invoke {@link   #lock ()} to   *  start the  Process of grabbing the lock; you may get the lock then  or it may be   *  some time later. <p/> you  can register a listener so that you are invoked   *   when you get the lock; otherwise you can ask if 
You have the lock  *  by calling {@link   #isOwner ()}  *  */ public class writelock extends protocolsupport {    private  Static finAl logger log = loggerfactory.getlogger (Writelock.class);
    private final String dir;
    private String id;
    private ZNodeName idName;
    private String ownerId;
    private String lastChildId;
    private byte[] data = {0x12, 0x34};
    private LockListener callback;
    private LockZooKeeperOperation zop;          /**      * zookeeper  Contructor for writelock      *  @param  zookeeper zookeeper  client instance      *  @param  dir the parent path  you want to use for locking      *  @param  acls the acls that you want to use for  all the paths,       * if null world read/
Write is used.      */    public writelock (ZooKeeper zookeeper,  STRING&NBSP;DIR,&NBSP;LIST&LT;ACL&GT;&NBSP;ACL)  {        super (
Zookeeper);
        this.dir = dir;         if  (acl != null)  {    
        setacl (ACL);         }         this.zop
 = new lockzookeeperoperation (); &NBSP;&NBSP;&NBSP;&NBSP}          /**      *  zookeeper Contructor for writelock with callback      *  @param   Zookeeper the zookeeper client instance      *  @param   Dir the parent path you want to use for locking       *  @param  acl the acls that you want to use for  all the paths      *  @param  callback the call  back instance      */    public writelock ( zookeeper zookeeper, string dir, list<acl> acl,               locklistener callback)  {    
    this (ZOOKEEPER,&NBSP;DIR,&NBSP;ACL);         this.callback = calLback; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * return the  Current locklistener      *  @return  the locklistener       */    public locklistener getlocklistener ()  {  
      return this.callback; &NBSP;&NBSP;&NBSP;&NBSP}          /**      *  register a different call back listener      * @ Param callback the call back instance      */     public void setlocklistener (locklistener callback)  {    
    this.callback = callback; &NBSP;&NBSP;&NBSP;&NBSP}     /** &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;*&NBSP;REMOVES&Nbsp;the lock or associated znode if       * you  no longer require the lock. this also       *  removes your request in the queue for locking     
 * in case you do not already hold the lock.      *  @throws  RuntimeException throws a runtime 
Exception      * if it cannot connect to zookeeper.      */    public synchronized void unlock ()   throws runtimeexception {                  if  (!isclosed ()  && id != null)  {         &nBsp;   // we don ' T need to retry this operation in  the case of failure              // as zk will remove ephemeral files and we don ' t  Wanna hang             // this  Process when closing if we cannot reconnect to zk              try {                 ZooKeeperOperation zopdel = new  Zookeeperoperation ()  {             
       public boolean execute ()  throws keeperexception,                          interruptedexception {                         zookeeper.delete (id, -1);                      
       return Boolean.TRUE;                      }                 }
;                 zopdel.execute
();             } catch  ( Interruptedexception e)  {                 log.warn ("caught: "  + e, e);                 //set 
That we have been interrupted.                thread.currentthread
(). interrupt ();             } catch  ( Keeperexception.nonodeexception e)  {                 // do nothing              } catch  (keeperexception e)  {      
          log.warn ("caught: "  + e, e);                 throw  (runtimeexception)  new runtimeexception (E.getmessage ()).                   
  initcause (e);             }              finally {                 if  (callback != null)  {                     
Callback.lockreleased ();                 }   
              id = null;             }         &nBSP;} &NBSP;&NBSP;&NBSP;&NBSP}          /**        * the watcher called on        * getting  watch while watching       * my predecessor       */    private class lockwatcher implements watcher  {        public void process (WatchedEvent event)  {            // lets either  Become the leader or watch the new/updated node              log.debug ("watcher fired on path: "  +  event.getpath ()  +  " state: "  +                        event.getstate ()  +  " type
  " + event.gettype ());             try {    
            lock ();             } catch  (Exception e)  {                
Log.warn ("failed to acquire lock: "  + e, e);             }        &NBSP;&NBSP}     }          /**       * a zoookeeper operation that is mainly responsible       * for all tHe magic required for locking.      */    private  class lockzookeeperoperation  implements zookeeperoperation {                  /** find if we have been created earler
 if not create our node          * 
         *  @param  prefix the prefix node          *  @param  zookeeper teh zookeeper  client          *  @param  dir the dir &NBSP;PARETN          *  @throws  keeperexception           *  @throws  interruptEdexception          */         private void findprefixinchildren (string prefix, zookeeper zookeeper,  String dir)               throws  keeperexception, interruptedexception {          
  list<string> names = zookeeper.getchildren (Dir, false);             for  (string name :  names)  {                 if  (Name.startswith (prefix))  {                     id = dir +  "/"  + 
Name                     if  (LOG.isDebugEnabled ())  {                         log.debug ("Found id created last time:
  " + id);                      }                 
    break;                 }              }              if  (id == null)  {                 id = zookeeper.Create (dir +  "/"  + prefix, data,                           getacl ()
,  ephemeral_sequential);                 if  ( Log.isdebugenabled ())  {             
       log.debug ("created id: "  + id);                 }   
          }         }                  /**           * the command that is run and  retried for actually           * obtaining the  lock          *  @return  if the command  was successful or not          */         public boolean execute ()  throws KeeperException, 
interruptedexception {            do {                 if  (ID  == null)  {              
      long sessionid = zookeeper.getsessionid ();                      string prefix =  "x"  + sessionId +  "-";                      // lets try look up the current id if we failed                        // in the middle of creating the znode                      
Findprefixinchildren (Prefix, zookeeper, dir);                   
  idname = new znodename (ID);                 }                  if  (id != null)  {                     List<String> names = 
Zookeeper.getchildren (Dir, false);                      if  (Names.isempty ())  {                         log.warn ("No children  in:  " + dir + "  when we ' ve just  " +                        
   "Created one! lets recreate it ...");                        &nbSp; // lets force the recreation of the id                        
  id = null;                      } else {                         // lets sort them  explicitly  (though they do seem to come back in order  ususally :)                          SortedSet<ZNodeName> sortedNames = 
New treeset<znodename> ();                         for  (String name :  Names)  {                             sortednames.add (New ZNodeName (dir
 +  "/"  + name));                          }                          ownerid = sortednames.first (
). GetName ();                          sortedset<znodename> lessthanme = sortednames.headset
(Idname);                         if  (!lessThanMe.isEmpty () )  {                             ZNodeName lastChildName = 
Lessthanme.last ();                   
          lastchildid = lastchildname.getname ();                              if  (log.isdebugenabled ())  {                                  log.debug ("Watching less than me node:  " + lastchildid);                              }                            
  stat stat = zookeeper.exists (Lastchildid, new lockwatcher ());                              if  (stat != null)  {                    
             return Boolean.FALSE;                             } else {                                  log.warn ("Could not find the"  +                                            " stats for less than me: "  + 
Lastchildname.getname ());                              }                          } else&nBsp {                             if  (Isowner ())  {                                  if  (callback != null)  {                                      
Callback.lockacquired ();                                  }                                  return boolean.true;                              }                          }   
                  }                 }              }         
    while  (Id == null);
            return Boolean.FALSE;
        }     };      /**      * Attempts to acquire the exclusive  Write lock returning whether or not it was      *  acquired. note that the exclusive lock may be acquired some  time later after      * this method has been 
Invoked due to the current lock owner going away.      */    public synchronized boolean lock ()  throws keeperexception, interruptedexception {         if  (isclosed ())  {            
return false;         }         
Ensurepathexists (dir);         return  (Boolean)  retryoperation (ZOP); &NBSP;&NBSP;&NBSP;&NBSP}     /**      * return the  Parent dir for lock      *  @return  the parent dir
 used for locks.      */    public string getdir ()  {  
      return dir; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * returns true  if this node is the owner of the      *   lock  (or the leader)      */    public  Boolean isowner ()  {        return id != null  && ownerid != null &&aMp; id.equals (ownerID); &NBSP;&NBSP;&NBSP;&NBSP}     /**      * return the  Id for this lock      *  @return  the id for 
This lock      */    public string getid ()  {
       return this.id; &NBSP;&NBSP;&NBSP;&NBSP}}

The

Notices the lock here, may fail, try multiple times, and sleep for a period of time after each failure.

Classes used to sort the secondary node size order:

/**  *  * Licensed to the Apache Software Foundation  (ASF)   Under one or more  * contributor license agreements.  see the  notice file distributed with  * this work for additional 
Information regarding copyright ownership.  * the asf licenses this file to you under the apache  license, version 2.0  *  (the  "License");  you may not use  this file except in compliance with  * the License.   You may obtain a copy of the license at  *  * http:// www.apache.org/licenses/LICENSE-2.0  *  * unless required by applicable law  or agreed to in writing, software  * distributed under the License is distributed on an  "As is"  basis,  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, 
Either express or implied.  * See the License for the specific language governing 
Permissions and  * limitations under the license.
 */package org.apache.zookeeper.recipes.lock;
import org.slf4j.logger;
import org.slf4j.loggerfactory; /**  * Represents an ephemeral znode name which has an  Ordered sequence number  * and can be sorted in order  *   * * class znodename implements comparable<znodename> {    private
 final String name;
    private String prefix;     private int sequence = -1;     private static final Logger LOG = 
Loggerfactory.getlogger (Znodename.class);          public znodename (string name)  {         if  (name == null)  {             throw new nullpointerexception ("Id cannot be
 null ");         }         this.name
 = name;
        this.prefix = name;
        int idx = name.lastindexof ('-');         if  (idx >= 0)  {             this.prefix = name.substring (0,&NBSP;IDX);             try {                 this.sequence = 
Integer.parseint (name.substring (idx + 1));                 // if  an exception occurred we misdetected a sequence suffix,                  // so return
 -1.             } catch  ( Numberformatexception e)  {            
    log.info ("number format exception for "  + idx, e);             } catch  (arrayindexoutofboundsexception e)  {                log.info ("Array out of bounds for
  " + idx, e);             }        &NBSP;&NBSP}     }      @Override     public 
String tostring ()  {        return name.tostring (); &NBSP;&NBSP;&NBSP;&NBSP}      @Override     public boolean  Equals (Object o)  {        if  (This == o)
 return true;         if  (o == null | |
 getclass ()  != o.getclass ()  return false;         znodename sequence =  (Znodename)  o;
        if  (!name.equals (sequence.name))  return false;
        return true; &NBSP;&NBSP;&NBSP;&NBSP}      @Override     public int 
Hashcode ()  {        return name.hashcode ()  + 37;
&NBSP;&NBSP;&NBSP;&NBSP}     public int compareto (ZNodeName that)  {         int answer = this.prefix.compareto (That.prefix)
;         if  (answer == 0)  {    
        int s1 = this.sequence;
            int s2 = that.sequence;             if  (s1 == -1 &&  s2 == -1)  {             
   return this.name.compareto (That.name);             }              answer = s1 == -1 ? 1 : s2 ==
 -1 ? -1 : s1 - s2;         }         return 
Answer &NBSP;&NBSP;&NBSP;&NBSP}     /**      * returns the  name of the znode      */    public  String getname ()  {        return name;     &NBSP}     /**      * Returns the sequence  Number      */    public int getznodename ()  { 
       return sequence; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * returns the  text prefix before the sequence number      */     public string getprefix ()  {        
return prefix; &NBSP;&NBSP;&NBSP;&NBSP}}

Zookeeper Unified Operation Zookeeperoperation Interface:

Public interface Zookeeperoperation {
/**
* Performs the operation-which of May involved Ple times If the connection
* to zookeeper closes during this operation
*
* @return The result The operation or null
* @throws keeperexception
* @throws interruptedexception * * *
PU Blic Boolean execute () throws Keeperexception, interruptedexception;
}

Because the zookeeper operation fails, this class encapsulates several attempts:

/**  *  * Licensed to the Apache Software Foundation  (ASF)   Under one or more  * contributor license agreements.  see the  notice file distributed with  * this work for additional 
Information regarding copyright ownership.  * the asf licenses this file to you under the apache  license, version 2.0  *  (the  "License");  you may not use  this file except in compliance with  * the License.   You may obtain a copy of the license at  *  * http:// www.apache.org/licenses/LICENSE-2.0  *  * unless required by applicable law  or agreed to in writing, software  * distributed under the License is distributed on an  "As is"  basis,  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, 
Either express or implied.  * See the License for the specific language governing 
Permissions and  * limitations under the license.
 */package org.apache.zookeeper.recipes.lock;
import org.slf4j.logger;
import org.slf4j.loggerfactory;
import org.apache.zookeeper.createmode;
import org.apache.zookeeper.keeperexception;
import org.apache.zookeeper.zoodefs;
import org.apache.zookeeper.zookeeper;
import org.apache.zookeeper.data.acl;
import org.apache.zookeeper.data.stat;
import org.apache.zookeeper.recipes.lock.zookeeperoperation;
import java.util.list;
import java.util.concurrent.atomic.atomicboolean; /**  *  A base class for protocol implementations which provides a number  of higher   * level helper methods for working with  Zookeeper along with retrying synchronous  *  operations if the  connection to ZooKeeper closes such as   *  {@link  # Retryoperation (zookeeperoperation)}  *  */class protocolsupport {     Private static final logger log = loggerfactory.getlogger (ProtocolSupport.class)
;
    protected final ZooKeeper zookeeper;
    private atomicboolean closed = new atomicboolean (FALSE);
    private long retryDelay = 500L;
    private int retryCount = 10;     private list<acl> acl = zoodefs.ids.open_acl_unsafe;     public protocolsupport (zookeeper zookeeper)  {    
    this.zookeeper = zookeeper; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * closes this
 strategy and releases any zookeeper resources; but keeps the
     *  zookeeper instance open      */     public void close ()  {        if   (Closed.compareandset (false, true))  {         
   doclose (); &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP}     }           /**      *  return zookeeper client instance      *  @return  zookeeper  client instance      */    public ZooKeeper  Getzookeeper ()  {        return zookeeper;     &NBSP}     /**      * return the acl its 
Using      *  @return  the acl.      */    public list<acl> getacl ()  { 
       return acl; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * set the acl        *  @param  acl the acl to set to       */    public void setacl (List<ACL>&NBSP;ACL)  {        this.acl = acl;     &NBSP}     /**      * get the retry delay  in milliseconds      *  @return  the retry delay       */    public long getretrydelay ()  {  
      return retryDelay; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * Sets the  Time waited between retry delays      *  @param   Retrydelay the retry delay      */    public  Void setretrydelay (long retrydelay)  {        
this.retrydelay = retrydelay;     }      /**      * Allow derived classes to perform       * some custom closing operations to release 
Resources      */    protected void doclose ()  { &NBSP;&NBSP;&NBSP;&NBSP}     /**      * perform the  given operation, retrying if the connection fails       *  @return  object. it needs to be cast to the callee ' s
 expected       * return type.      */    protected object retryoperation ( zookeeperoperation operation)           throws  keeperexception, interruptedexception {        keeperexception exception = null;         for  (int i = 0; i < 
retrycount; i++)  {            try {                 return 
Operation.execute ();             } catch  ( Keeperexception.sessionexpiredexception e)  {                 log.warn ("session expired for: "  + 
zookeeper +  " so reconnecting due to: "  + e, e);
                throw e;            &nBSP;}  catch  (keeperexception.connectionlossexception e)  {                 if  (exception == null)  {                    
 exception = e;                 }                  log.debug ("Attempt "  + i +  " failed with connection loss so "  +                      
     "attempting to reconnect: "  + e, e);                 retrydelay (i);             }       
  }         throw exception; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * ensures that  the given path exists with no data, the current   
   * acl and no flags      *  @param  path      */    protected void ensurepathexists (String  Path)  {        ensureexists (path, null, acl, 
Createmode.persistent); &NBSP;&NBSP;&NBSP;&NBSP}     /**      * ensures that  the given path exists with the given data, ACL and  Flags &NBSP;&NBSP;&NBsp;  *  @param  path      *  @param  acl       *  @param  flags      */    protected  Void ensureexists (Final string path, final byte[] data,              final List<ACL> acl, final  createmode flags)  {        try {   
         retryoperation (New zookeeperoperation ()  {                 public  Boolean execute ()  throws keeperexception, interruptedexception {                     stat  stat = zookeeper.exIsts (Path, false);                      if  (stat != null)  {         
               return true;                      }                 
    zookeeper.create (Path, data, acl, flags);                   
  return true;                 }   
          });         }  catch  (keeperexception e)  {          
  log.warn ("caught: "  + e, e);         } catch  (interruptedexception e)  {             log.warn ("caught: "  + e,
&NBSP;E);              }     /**       * Returns true if this protocol has been  Closed      *  @return  true if this protocol is 
Closed      */    protected boolean isclosed ()  {
        return closed.get (); &NBSP;&NBSP;&NBSP;&NBSP}     /**      * perForms a retry delay if this is not the first attempt       *  @param  attemptcount the number of the attempts  performed so far      */    protected void  retrydelay (Int attemptcount)  {        if  ( attemptcount > 0)  {             try {                
Thread.Sleep (Attemptcount * retrydelay);             } catch  ( Interruptedexception e)  {            
    log.debug ("failed to sleep: "  + e, e);              }          }    &nbsp}}

This class is the interface that the client triggers when it acquires lock and release lock:

Public interface Locklistener {
/** * Call back called when the
lock
* is acquired
*/<    c4/> public void lockacquired ();
/**
* Call back called the Lock is
* released.
*
/public void lockreleased ();
}

Queue (queue)

Distributed queues are common data structures, in order to implement distributed queues in zookeeper, you first need to specify a Znode node as the queue node node), where each distributed client puts the data into the queue by calling the Create () function, and when the Create () is called, it ends with a "qn-" and sets the order and temporary (sequence and ephemeral) node flags. The new pathname has the following string pattern because of the order flag of the node: "_path-to-queue-node_/qn-x", and X is the only self-added number. Clients that need to remove data from the queue call the GetChildren () function first, while the watch is set to true on the queue node (\ node) and the node that handles the smallest number (that is, the data is fetched from the smallest number of nodes). The client does not need to call GetChildren () again, and the data in the queue is fetched. If there are no child nodes in the queue node, the client that reads the queue needs to wait for the queue's monitoring event notification. The implementation steps for

are basically as follows:

Prerequisite: Need a queue root node dir

Join: Create a node by creating (), put the shared data on the node, the node type is Persistent_ Sequential, temporary, sequential.

out of line: Because the queue may be empty, 2 ways to handle it: a wait waiting for a null, a return exception.

Wait by: Here the Countdownlatch wait and watcher notification mechanism is used to obtain the smallest data (FIFO) of the node order using the TreeMap sort.

Throws an exception: GetChildren () when the queue data is fetched, if size==0 throws an exception.

Implementation of a distributed queue, detailed code:

/**  *  * Licensed to the Apache Software Foundation  (ASF)   Under one or more  * contributor license agreements.  see the  notice file distributed with  * this work for additional 
Information regarding copyright ownership.  * the asf licenses this file to you under the apache  license, version 2.0  *  (the  "License");  you may not use  this file except in compliance with  * the License.   You may obtain a copy of the license at  *  * http:// www.apache.org/licenses/LICENSE-2.0  *  * unless required by applicable law  or agreed to in writing, software  * distributed under the License is distributed on an  "As is"  basis,  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, 
Either express or implied.  * See the License for the specific language governing 
Permissions and  * limitations under the license.
 */package org.apache.zookeeper.recipes.queue;
import java.util.list;
import java.util.nosuchelementexception;
import java.util.treemap;
import java.util.concurrent.countdownlatch;
import org.slf4j.logger;
import org.slf4j.loggerfactory;
import org.apache.zookeeper.createmode;
import org.apache.zookeeper.keeperexception;
import org.apache.zookeeper.watchedevent;
import org.apache.zookeeper.watcher;
import org.apache.zookeeper.zoodefs;
import org.apache.zookeeper.zookeeper; Import&nBsp;org.apache.zookeeper.data.acl;
import org.apache.zookeeper.data.stat; /**  *   * a <a href= "package.html" >protocol to implement a
 distributed queue</a>.  *   */public class distributedqueue {    private static
 final logger log = loggerfactory.getlogger (Distributedqueue.class);
    private final String dir;
    private ZooKeeper zookeeper;
    private List<ACL> acl = ZooDefs.Ids.OPEN_ACL_UNSAFE;
    private final String prefix =  "qn-";     public distributedqueue (zookeeper zookeeper, string dir, list &LT;ACL&GT;&NBSP;ACL) {        this.dir = dir;        &nbsP; if (acl != null) {            
this.acl = acl;         }         
this.zookeeper = zookeeper;                  //Add  Root dir first if not exists         if  ( Zookeeper != null)  {            try  {                stat
 s = zookeeper.exists (Dir, false);                 if  (S  == null)  {                  &nbSp;  zookeeper.create (dir, new byte[0], acl, createmode.persistent);                 }   
          } catch  (keeperexception e)  {                 log.error (
E.tostring ());             } catch  ( Interruptedexception e)  {            
    log.error (E.tostring ());             }        &NBSP;&NBSP}     }     /**      * returns
 a map of the children, ordered by id.      *  @param  watcher optional watcher on getchildren ()  
Operation.      *  @return  map from id to child name for  all children      */    private treemap<long, String> orderedchildren (Watcher watcher)  throws KeeperException,  interruptedexception {        TreeMap<Long,String> 
Orderedchildren = new treemap<long,string> ();
        List<String> childNames = null;         try{         
   childnames = zookeeper.getchildren (Dir, watcher);         }catch  (keeperexception.nonodeexception e) {            throw e; &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP}         for (String  childname : childnames) {             try{                // Check format                  if (!childname.regionmatches (0, prefix, 0, prefix.length ())) {                     log.warn ("
found child node with improper name:  " + childname);                   
  continue;                 }           
      string suffix = childname.substring (Prefix.length ());                 Long 
Childid = new long (suffix);                 
Orderedchildren.put (Childid,childname);
            }catch (NumberFormatException e) {                 log.warn (" found child node with improper format :  " + childName +
  " "  + e,e);             }          }         return orderedchildren; &NBSP;&NBSP;&NBSP;&NBSP}     /**      * Find the 
Smallest child node.      *  @return  the name of the smallest child
 node.      */    private string smallestchildname ()   throws keeperexception, interruptedexception {        
long minid = long.max_value;
        String minName =  "";
        List<String> childNames = null;         try{         
   childnames = zookeeper.getchildren (Dir, false);         }catch (KEeperexception.nonodeexception e) {           
 log.warn ("caught: "  +e,e);
            return null; &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP}         for (String  childname : childnames) {             try{                // Check format                  if (!childname.regionmatches (0, prefix, 0, prefix.length ())) {                     log.warn ("
found child node with improper name:  " + childname);                     continue;                 }                  string suffix =
 childname.substring (Prefix.length ());                 long 
Childid = long.parselong (suffix);                 if (childId  < minid) {              
      minId = childId;                   
  minName = childName;                  }              }catch (numberformatexception e) {                 log.warn ("FOUND&NBSP;CHILD&NBSP;NODE&NBSP;WITH&NBSP;IMPROPER&NBSP;FORMAT&NBSP;:
  " + childName + "   " + e,e);             }        &NBSP;&NBSP}         if (minid < long.max_value) { 
           return minName;         }else{         
   return null;              }     /**       * return&nbSp;the head of the queue without modifying the queue.      *  @return  the data at the head of the
 queue.      *  @throws  nosuchelementexception      * @ Throws keeperexception      *  @throws  interruptedexception       */    public byte[] element ()  throws  nosuchelementexception, keeperexception, interruptedexception {     
   TreeMap<Long,String> orderedChildren;         // element, take, and remove follow
 the same pattern.         // we want to return the child  node with the smallesT sequence number.         // since other clients are remove () Ing and take () ing nodes concurrently,           // the child with the smallest sequence number in 
Orderedchildren might be gone by the time we check.         // we don ' T call getchildren again  until we have tried the rest of the nodes in sequence
 order.         while (true) {             try{             
   orderedchildren = orderedchildren (NULL);         &nBsp;   }catch (keeperexception.nonodeexception e) {      
          throw new nosuchelementexception ();             }              if (Orderedchildren.size ()  == 0 )  throw new 
Nosuchelementexception ();             for (String headNode :  Orderedchildren.values ()) {                 if (headnode != null) {                     try{                         return&nbsP;zookeeper.getdata (dir+ "/" +headnode, false, null);                      }catch (keeperexception.nonodeexception e) {                         //another  client removed the node first, try next                      }                  }              }              &NBSP}     /**      * attempts to remove the
 head of the queue and return it.     *  @return  the former head of the queue       *  @throws  nosuchelementexception      *  @throws  keeperexception      *  @throws  interruptedexception       */    public byte[] remove ()  throws nosuchelementexception ,  keeperexception, interruptedexception {        
treemap<long,string> orderedchildren;         // same as for element.  should
 refactor this.         while (true) {             try{                 orderedchildren = ordereDchildren (NULL);             }catch (KeeperException.NoNodeException  e) {                throw
 new nosuchelementexception ();             }              if (Orderedchildren.size ()  == 0)  throw new 
Nosuchelementexception ();             for (String headNode :  Orderedchildren.values ()) {              
  string path = dir + "/" +headnode;                 try{                     byte[] data = zookeeper.getdata (Path, false, null);                   
  zookeeper.delete (path, -1);                   
  return data;                 }catch ( Keeperexception.nonodeexception e) {                     // Another client deleted the 
Node first.                 }   
          }         }     }     private class latchchildwatcher implements watcher {     
   CountDownLatch latch;         public latchchildwatcher () {     
       latch = new countdownlatch (1);         }         public  Void process (watchedevent event) {             log.debug ("watcher fired on path: "  + event.getpath ()  +  "  state:  " +                       event.getstate ()  +  " type "  + event.gettype (
));
            latch.countdown ();         }         public void  await ()  throws interruptedexception {        
    latch.await ();              }     /**       * removes the head of the queue and returns
 it, blocks until it succeeds.      *  @return  the former head of the queue       *  @throws  nosuchelementexception      *  @throws  keeperexception      *  @throws  interruptedexception       */    public byte[] take ()  throws KeeperException,  Interruptedexception {        treemap<long,string> orderedchildren;         // same as for element.  should
 refactor this.         while (true) {        
    latchchildwatcher childwatcher = new latchchildwatcher ();             try{                 orderedchildren = orderedchildren (
Childwatcher);             }catch (KeeperException.NoNodeException  e) {                
Zookeeper.create (dir, new byte[0], acl, createmode.persistent);                 continue;             }              if (Orderedchildren.size ()  == 0) {     
           childwatcher.await ();
                continue;             }              for (String headnode : orderedchildren.values ()) {                 string path
 = dir + "/" +headnode;                 try{                     byte[] data = 
Zookeeper.getdata (Path, false, null);                   
  zookeeper.delete (path, -1);                   
  return data;                 }catch ( Keeperexception.nonodeexception e) {                     // Another client deleted the 
Node first.                 }   
          }         } &NBSP;&NBSP;&NBSP;&NBSP}     /**      * inserts data
 into queue.      *  @param  data      *  @return  true  if data was successfully added      */     public boolean offer (Byte[] data)  throws KeeperException,  interruptedexception{        for (;;) {            try{                 zookeeper.create (dir+ "/" +prefix, data,
 acl, createmode.persistent_sequential);                 return 
True             }catch (KeeperException.NoNodEexception e) {               
 zookeeper.create (dir, new byte[0], acl, createmode.persistent);             }        &NBSP;&NBSP}     }     /**      * returns  the data at the first element of the queue, or null
 if the queue is empty.      *  @return  data at the first element of 
The queue, or null.      *  @throws  keeperexception      *  @throws  interruptedexception      */    public byte[] peek ()  throws keeperexception, interruptedexception{        try{        
    return element ();         }catch (nosuchelementexception e) {    
        return null;              }     /**       * attempts to remove the head of the queue
 and return it. returns null if the queue is empty.
     *  @return  head of the queue or null.      *  @throws  keeperexception      *  @throws  interruptedexception      */    public byte[] poll ()  throws keeperexception, interruptedexception {        try{  
          return remove ();         }catch (nosuchelementexception e) {    
        return null;         }    &nbsp}}

based on zookeeper simple implementation of distributed locks

uses the ephemeral_sequential type node and watcher mechanism of zookeeper. To simply implement distributed locks.
Main idea:
1, open 10 threads, create each ephemeral_sequential node named sub under the Dislocks node,
2, get all the child nodes under the Dislocks node, sort, if your own node number is the smallest, Gets the lock;
3, otherwise watch in front of its own node, listening to its deletion, into the 2nd step (re-detection of the sorting is to prevent the connection failure of the listening node, resulting in the deletion of the node);
4, delete itself sub nodes, release the connection;

Here's the interrupt. 4 node types for zookeeper:

    public enum CreateMode {                      /**            *  Persistent node: Once the node is created, it will always exist and will not be deleted because the client session fails;            */           PERSISTENT  (0, false ,  false),                   /**          *  Persistent order nodes: The basic features are consistent with the persistent nodes, and the process of creating the nodes Zookeeper will automatically append a monotonically growing number suffix after its name, as the new node name;           */            PERSISTENT_SEQUENTIAL  (2, false, true),
                  /**           *   Temporary node: After a client session fails or the connection is closed, the node is automatically deleted and no child nodes can be created under the temporary node. Otherwise reported the following error:org.apache.zookeeper.keeperexception$nochildrenforephemeralsexception;            */           EPHEMERAL  (1,  true, false),                   /**           *  Temporary sequential nodes: The basic attributes are consistent with the temporary node, and in the process of creating a node, Zookeeper will automatically append a monotonically growing number suffix after its name, as the new node name;            */            EPHEMERAL_SEQUENTIAL  (3, true, true );           private static final Logger  Log = loggerfactory.getlogger (createmode.class);            private boolean ephemeral;           private boolean sequential;            private int flag;            createmode (int flag, boolean ephemeral, boolean  Sequential)  {                this.flag = flag;                this.ephemeral = ephemeral;                this.sequential = sequential;            }           public boolean isephemeral ( )  {               return  ephemeral;     &Nbsp;     }           public  Boolean issequential ()  {             
  return sequential;           }           public int toflag ()  {                return flag;            }           static public  createmode fromflag (int flag)  throws KeeperException {                switch (flag)  {                case 0: return CreateMode.PERSISTENT;     &NBsp;          case 1: return  createmode.ephemeral;               case  2: return CreateMode.PERSISTENT_SEQUENTIAL;                case 3: return CreateMode.EPHEMERAL_SEQUENTIAL ;                default:                    log.error ("Received  An invalid flag value to convert to a createmode ");                    throw new  Keeperexception.badargumentsexception ();                }  
        }       } 

Test code:

    package zookeeper;       import  org.slf4j.logger;       import org.slf4j.LoggerFactory;        import org.apache.zookeeper.*;       import 
org.apache.zookeeper.data.stat;       import java.util.List;       import java.io.IOException;       import  java.util.collections;       import java.util.concurrent.CountDownLatch;               public class distributedlock  implements Watcher{           private int  threadId;           PRIVATE ZOOKEEPER ZK  = null;           private string selfpath;            private String waitPath;            private String LOG_PREFIX_OF_THREAD;            private static final int SESSION_TIMEOUT = 10000;            private static final String GROUP_PATH =  "/ Dislocks ";           private static final  string sub_path =  "/dislocks/sub";           
private static final string connection_string =  "192.168.*.*:2181";                       Private static final int thread_num = 10;             //ensure the connection ZK succeeds;           private CountDownLatch  Connectedsemaphore = new countdownlatch (1);            //ensure all threads run over;           private static 
Final countdownlatch threadsemaphore = new countdownlatch (THREAD_NUM);           private static final logger log =  loggerfactory.getlogger (allzookeeperwatcher.class);            public distributedlock (int id)  {                this.threadId = id;                log_prefix_of_thread =  "" "+threadid+" "";            }           public static  Void main (String[] args)  {                for (int i=0; i < thread_num; i++) {                    final int threadid  = i+1;                    new thread () {                         @Override                         public  Void run ()  {                            try{                                 distributedlock dc = new distributedlock (threadId);                                 dc.createconnection (connection_string, session_timeout);                     If             //group_path does not exist, it can be created by a thread;                                 synchronized  (threadsemaphore) {                                     dc.createpath (GROUP_PATH,  "This node by thread"  + threadId +  "Create",  true);                                 }                                dc.getlock ();                             } catch  (exception e) {                                 log.error ("" "+threadid+" "Thread"   thrown exception: ");                                 e.printstacktrace ();                            }                        }                    }.
Start ();               }               try {        &nbsP;           threadsemaphore.await ();                    log.info ("All threads run over!");                } catch  ( Interruptedexception e)  {                    e.printstacktrace ();                }           }            /**           *   Get locks            *  @return             */           private  void getlOck ()  throws KeeperException, InterruptedException {                selfpath = zk.create (SUB_PATH,null,  zoodefs.ids.open_acl_unsafe, createmode.ephemeral_sequential);                log.info (log_prefix_of_thread+ "Create Lock Path:" +selfpath);                if (Checkminpath ()) {                    getlocksuccess ();                }            }           /**            *  Create nodes             *  @param  path  node path           *  @param  data  Initial data content            *  @return             */           public  Boolean createpath ( string path, string data, boolean needwatch)   throws keeperexception, interruptedexception {                if (zk.exists (path, needwatch) ==null) {                    log.info ( LOG_PREFIX_OF_ thread +  "Node creation success, path: "                             +  This.zk.create (&NBSP;PATH,&NBSP;&NBsp                          data.getbytes (),                             zoodefs.ids.open_acl_unsafe,                            CreateMode.PERSISTENT )                             +  ", content: "  + data );                }                return true;            }           /**            *  Create ZK link            *  @param   CONNECTSTRING&NBSP;&NBSP;ZK server address List            *  @param  sessiontimeout session Timeout            */            public void createconnection ( String  connectString, int sessionTimeout )  throws IOException,  interruptedexception {                    zk = new zookeeper ( connectstring, sessiontimeout,  this);                    connectedsemaphore.await ();           }           /* *           *  Get lock success            */           public void  Getlocksuccess ()  throws KeeperException, InterruptedException {                if (zk.exists (this.selfpath,false)  ==  NULL) {                    log.error (log_prefix_of_thread+ "This node is gone ...");                    return;                }                log.info (Log_prefix_of_thread +  "Get lock success, work quickly!" ");               thread.sleep (2000);                log.info (LOG_PREFIX_OF_ thread +  "Delete this node:" +selfpath;                zk.delete (this.selfpath, -1);                releaseconnection ();                threadsemaphore.countdown ();           }            /**        
   *  close ZK link            */           public void releaseconnection () &nbsP {               if  (&NBSP;THIS.ZK  !=null )  {                    try {                        this.zk.close ();                    } catch  (  interruptedexception e )  {}                }                Log.info (log_prefix_of_thread +  "Release Connection");            }           /**            *  Check yourself isNot the smallest node            *  @return             */           public 
Boolean checkminpath ()  throws KeeperException, InterruptedException {                list<string> subnodes  = zk.getchildren (group_path, false);                 collections.sort (subnodes);                 int index = subnodes.indexof (  Selfpath.substring (Group_path.length () +1);                 switch  (index) {                     case -1:{                         log.error (log_prefix_of_thread+ "This node is gone ..." + Selfpath);                         return false;                     }                     case 0:{                          log.info (log_prefix_of_thread+ "Child node, I really am the eldest" +selfpath);                         return true;    &NBSP;&NBSP;&NBsp;              }                     default:{                          this.waitpath = group_path + "/" + subnodes.get (index -  1);                 
       log.info (log_prefix_of_thread+ "Get the +waitpath) in the child node, in front of me;                         try{                             zk.getdata (WaitPath, true,  new stat ());  &nbsp                           return false;                         }catch ( Keeperexception e) {                             if (zk.exists (WaitPath,false)  == null) {                                 log.info (LOG_ Prefix_of_thread+ "Child node, in front of me," +waitpath+ "has disappeared, happiness came too suddenly?");                                  reTurn checkminpath ();                             }else{                                  throw e;                              }                         }                     }                                          }                        }             @Override            public void  Process (watchedevent event)  {                if (event == null) {                    return;                }                event.keeperstate keeperstate = event.getstate ();                event.eventtype eventtype = event.gettype ();                if  ( event.keeperstate.syncconnected  == keeperstate)  {                    if  ( Event.EventType.None == eventType )  {                         log.info ( LOG_PREFIX_OF_THREAD +  "Successfully connected to ZK Server"  );                         connectedsemaphore.countdown ();                    }else if  (Event.gettype ()  ==  Event.eventtype.nodedeleted && event.getpath (). Equals (Waitpath)  {             &nBsp;          log.info (log_prefix_of_thread +  " I got the information, the guy in front of me is dead. ");                        try {                            if (CheckMinPath ()) {                                 getlocksuccess ();                             }                        } catch  (keeperexception e)  { & NBsp                          e.printstacktrace ();                        } catch  ( Interruptedexception e)  {                            e.printstacktrace ();                         }                    }               }else  if  ( Event.KeeperState.Disconnected == keeperState )  {                    log.info ( log_prefix_of_thread +   Disconnect from ZK server  );                } else if  ( Event.KeeperState.AuthFailed == keeperState )  {                    log.info ( LOG_PREFIX_OF_THREAD +  permission check failed  );                } else if  ( Event.KeeperState.Expired ==  keeperstate )  {                    log.info ( LOG_PREFIX_OF_THREAD +  "session invalidation"  );                }            } &nbSp     }

Log configuration file:

# default    log4j.rootlogger=info,console      #   # Log
 INFO level and above messages to the console   #   log4j.appender.console=org.apache.log4j.consoleappender   log4j.appender.console.threshold=info 
  log4j.appender.console.layout=org.apache.log4j.patternlayout   log4j.appender.console.layout.conversionpattern=%d{iso8601} - %m%n       
  log4j.appender.commonstat=org.apache.log4j.dailyrollingfileappender   log4j.appender.commonstat.threshold=info   log4j.appender.commonstat.file=/home/zookeeper/ zookeeper-test-agent/logs/test.log   log4j.appender.commonstat.datepattern= '. '
yyyy-mm-dd      log4j.appender.commonstat.layout=org.apache.log4j.patternlayout   Log4j.appender.commonstat.layout.conversionpattern=[%d{yyyy-mm-dd hh:mm:ss}] - %m%n      log4j.logger.org.displaytag=warn   log4j.logger.org.apache.zookeeper=
error   log4j.logger.org.springframework=warn   log4j.logger.org.i0itec=warn   Log4j.logger.commonstat=info,commonstat

Run Result:

2014-11-19 11:34:10,894-"9th thread" successfully connected to ZK server
2014-11-19 11:34:10,895-"8th thread" successfully connected to ZK server
2014-11-19 11:34:10,894-"1th thread" successfully connected to ZK server
2014-11-19 11:34:10,894-"7th thread" successfully connected to ZK server
2014-11-19 11:34:10,894-"4th thread" successfully connected to ZK server
2014-11-19 11:34:10,895-"5th thread" successfully connected to ZK server
2014-11-19 11:34:10,896-"2nd thread" successfully connected to ZK server
2014-11-19 11:34:10,894-"10th thread" successfully connected to ZK server
2014-11-19 11:34:10,894-"3rd thread" successfully connected to ZK server
2014-11-19 11:34:10,895-"6th thread" successfully connected to ZK server
2014-11-19 11:34:10,910-"Thread 9th" node created successfully, Path:/dislocks, Content: This node is created by thread 9
2014-11-19 11:34:10,912-"Thread 9th" creates a lock path:/dislocks/sub0000000000
2014-11-19 11:34:10,917-"Thread 6th" creates a lock path:/dislocks/sub0000000001
2014-11-19 11:34:10,917-"9th Thread" child node, I really am the boss/dislocks/sub0000000000
2014-11-19 11:34:10,921-"Thread 3rd" creates a lock path:/dislocks/sub0000000002
2014-11-19 11:34:10,922-"6th thread" gets the child node, the/dislocks/sub0000000000 in front of me
2014-11-19 11:34:10,923-"9th thread" get lock success, work quickly!
2014-11-19 11:34:10,924-"thread 10th" creates a lock path:/dislocks/sub0000000003
2014-11-19 11:34:10,924-"3rd thread" gets the child node, the/dislocks/sub0000000001 in front of me
2014-11-19 11:34:10,928-"10th thread" gets the child node, the/dislocks/sub0000000002 in front of me
2014-11-19 11:34:10,929-"Thread 1th" creates a lock path:/dislocks/sub0000000004
2014-11-19 11:34:10,932-"Thread 5th" creates a lock path:/dislocks/sub0000000005
2014-11-19 11:34:10,935-"1th thread" gets the child node, the/dislocks/sub0000000003 in front of me
2014-11-19 11:34:10,936-"Thread 2nd" creates a lock path:/dislocks/sub0000000006
2014-11-19 11:34:10,936-"5th thread" gets the child node, the/dislocks/sub0000000004 in front of me
2014-11-19 11:34:10,940-"Thread 4th" creates a lock path:/dislocks/sub0000000007
2014-11-19 11:34:10,941-"2nd thread" gets the child node, the/dislocks/sub0000000005 in front of me
2014-11-19 11:34:10,943-"Thread 8th" creates a lock path:/dislocks/sub0000000008
2014-11-19 11:34:10,944-"4th thread" gets the child node, the/dislocks/sub0000000006 in front of me
2014-11-19 11:34:10,945-"Thread 7th" creates a lock path:/dislocks/sub0000000009
2014-11-19 11:34:10,946-"8th thread" gets the child node, the/dislocks/sub0000000007 in front of me
2014-11-19 11:34:10,947-"7th thread" gets the child node, the/dislocks/sub0000000008 in front of me
2014-11-19 11:34:12,923-"9th thread" Delete this node:/dislocks/sub0000000000
2014-11-19 11:34:12,926-"6th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:12,928-"6th Thread" child node, I really am the boss/dislocks/sub0000000001
2014-11-19 11:34:12,930-"9th thread" free connection
2014-11-19 11:34:12,930-"6th thread" get lock success, work quickly!
2014-11-19 11:34:14,930-"6th thread" Delete this node:/dislocks/sub0000000001
2014-11-19 11:34:14,937-"3rd thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:14,941-"3rd Thread" child node, I really am the boss/dislocks/sub0000000002
2014-11-19 11:34:14,943-"6th thread" free connection
2014-11-19 11:34:14,946-"3rd thread" get lock success, work quickly!
2014-11-19 11:34:16,946-"3rd thread" Delete this node:/dislocks/sub0000000002
2014-11-19 11:34:16,949-"10th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:16,951-"10th thread" child node, I really am the boss/dislocks/sub0000000003
2014-11-19 11:34:16,953-"3rd thread" free connection
2014-11-19 11:34:16,953-"10th thread" get lock success, work quickly!
2014-11-19 11:34:18,953-"10th thread" Delete this node:/dislocks/sub0000000003
2014-11-19 11:34:18,957-"1th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:18,960-"10th thread" free connection
2014-11-19 11:34:18,961-"1th thread" child node, I really am the boss/dislocks/sub0000000004
2014-11-19 11:34:18,964-"1th thread" get lock success, work quickly!
2014-11-19 11:34:20,964-"1th thread" Delete this node:/dislocks/sub0000000004
2014-11-19 11:34:20,967-"5th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:20,969-"5th Thread" child node, I really am the boss/dislocks/sub0000000005
2014-11-19 11:34:20,971-"1th thread" free connection
2014-11-19 11:34:20,971-"5th thread" get lock success, work quickly!
2014-11-19 11:34:22,971-"5th thread" Delete this node:/dislocks/sub0000000005
2014-11-19 11:34:22,974-"2nd thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:22,978-"2nd Thread" child node, I really am the boss/dislocks/sub0000000006
2014-11-19 11:34:22,979-"5th thread" free connection
2014-11-19 11:34:22,981-"2nd thread" get lock success, work quickly!
2014-11-19 11:34:24,981-"2nd thread" Delete this node:/dislocks/sub0000000006
2014-11-19 11:34:24,985-"4th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:24,989-"2nd thread" free connection
2014-11-19 11:34:24,989-"4th Thread" child node, I really am the boss/dislocks/sub0000000007
2014-11-19 11:34:24,995-"4th thread" get lock success, work quickly!
2014-11-19 11:34:26,995-"4th thread" Delete this node:/dislocks/sub0000000007
2014-11-19 11:34:26,998-"8th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:27,000-"8th Thread" child node, I really am the boss/dislocks/sub0000000008
2014-11-19 11:34:27,004-"8th thread" get lock success, work quickly!
2014-11-19 11:34:27,004-"4th thread" free connection
2014-11-19 11:34:29,004-"8th thread" Delete this node:/dislocks/sub0000000008
2014-11-19 11:34:29,007-"7th thread" received the information, platoon I front of the guy has hung up, I can not get out?
2014-11-19 11:34:29,009-"7th Thread" child node, I really am the boss/dislocks/sub0000000009
2014-11-19 11:34:29,010-"8th thread" free connection
2014-11-19 11:34:29,011-"7th thread" get lock success, work quickly!
2014-11-19 11:34:31,011-"7th thread" Delete this node:/dislocks/sub0000000009
2014-11-19 11:34:31,017-"7th thread" free connection
2014-11-19 11:34:31,017-All threads run over!