Zookeeper Learning Note 11 Zookeeper typical application scenario--distributed queue

This Learning resource is derived from the principles and practices of distributed consistency from Paxos to zookeeper, electronic publishing house

Distributed queues are simply two categories: one is a regular first-in, first-out queue, and the other is to wait until the queue element is clustered before the barrier model (synchronization queue) is scheduled to be executed uniformly. FIFO: First in, first out

The first-in first-out algorithm, with its simple and clear characteristics, is widely used in all aspects of computer science. The FIFO queue is also a very typical and widely applied sequential queue model, and the subsequent request is not processed until the request operation that enters the queue is completed first.

Implementing a FIFO queue with zookeeper is very similar to the implementation of a shared lock. FIFO queue is similar to a full-write shared lock model, the general design idea:
All clients will create a temporary order node under the/queue_fifo node, such as/queue_fifo/192.168.0.1-00000001,
After the nodes are created, the order of execution is determined by calling the GetChildren () interface to get all the child nodes under the/queue_fifo node, that is, to get all the elements in the queue to determine the order of their own node numbers in all child nodes If you are not a sub-node with the smallest ordinal number, then you need to enter the wait and register the watcher listener with the last node that is smaller than its ordinal number. After receiving the WATHCER notification, repeat steps 1

code example:

Distributedqueue

Package zookeeper;
Import Java.io.ByteArrayInputStream;
Import Java.io.ByteArrayOutputStream;
Import Java.io.ObjectInputStream;
Import Java.io.ObjectOutputStream;
Import java.util.Collections;
Import Java.util.Comparator;
Import java.util.List;

Import Java.util.concurrent.CountDownLatch; import org.
I0Itec.zkclient.ExceptionUtil; import org.
I0Itec.zkclient.exception.ZkNoNodeException;
Import Org.apache.zookeeper.CreateMode;
Import org.apache.zookeeper.WatchedEvent;
Import Org.apache.zookeeper.Watcher;
Import Org.apache.zookeeper.Watcher.Event.EventType;
Import Org.apache.zookeeper.ZooDefs;
Import Org.apache.zookeeper.ZooKeeper;
Import Org.apache.zookeeper.data.Stat;
Import Org.slf4j.Logger;

Import Org.slf4j.LoggerFactory; /** * Distributed queue, implementation of synchronization queue * * @author Linbingwen * * @param <T> * * public class Distributedqueue<t> {pri

    vate static Logger Logger = Loggerfactory.getlogger (Distributedqueue.class); Protected final ZooKeeper zookeeper;//for Operation ZooKeeper Cluster ProTected Final String root;//represents the root node private int queuesize;

    Private String Startpath = "/queue/start"; Protected static final String Node_name = "n_";//the name of the sequential node public distributedqueue (ZooKeeper ZooKeeper, String root, I
        NT queuesize) {this.zookeeper = ZooKeeper;
        This.root = root;
        This.queuesize = queuesize;
    Init (); }/** * Initialize root directory */private void Init () {try {stat stat = zookeeper.exists (root, FA LSE);//Determine if the root directory exists if (stat = = null) {zookeeper.create (root, NULL, ZooDefs.Ids.OPEN_ACL_UNSA
            FE, createmode.persistent); } zookeeper.delete (Startpath,-1);
        Delete queue full flag} catch (Exception e) {logger.error ("Create RootPath Error", E); }}/** * Gets the size of the queue * * @return * @throws Exception */public int size () throws Excep
    tion {return Zookeeper.getchildren (root, False). Size ();

}    /** * Determine if the queue is empty * * @return * @throws Exception */public Boolean isEmpty () throws Except
    Ion {return Zookeeper.getchildren (root, False). Size () = = 0; }/** * Bytes to Object * * @param bytes * @return */private Object bytetoobject (byte[]
        bytes) {Object obj = null;
            try {//ByteArray to object Bytearrayinputstream bi = new Bytearrayinputstream (bytes);

            ObjectInputStream oi = new ObjectInputStream (BI);
            obj = Oi.readobject ();
            Bi.close ();
        Oi.close ();
            } catch (Exception e) {logger.error ("translation" + E.getmessage ());
        E.printstacktrace ();
    } return obj; }/** * Object to BYTE * * @param obj * @return */private byte[] Objecttobyte (Java.lang.
        Object obj) {byte[] bytes = NULL;
          try {//object to ByteArray  Bytearrayoutputstream bo = new Bytearrayoutputstream ();
            ObjectOutputStream oo = new ObjectOutputStream (bo);

            Oo.writeobject (obj);

            bytes = Bo.tobytearray ();
            Bo.close ();
        Oo.close ();
            } catch (Exception e) {logger.error ("translation" + E.getmessage ());
        E.printstacktrace ();
    } return bytes;
     /** * Provides data to the queue, the queue full will block wait until the start flag is cleared * * @param element * @return * @throws Exception */Public Boolean offer (T Element) throws Exception {//The full path to the build data node String Nodefullpath = Root.conc
        at ("/"). Concat (Node_name); try {if (Queuesize > Size ()) {//Create persistent node, write Data zookeeper.create (Nodefullpa
                Th, Objecttobyte (Element), ZooDefs.Ids.OPEN_ACL_UNSAFE, createmode.persistent);
                  Then determine if the queue is full if (Queuesize > Size ()) {  Zookeeper.delete (Startpath,-1); Make sure that it does not exist} else {zookeeper.create (Startpath, NULL, ZooDefs.Ids.OPEN_ACL_UNSAFE, creat
                Emode.persistent);
                    }} else {//Create a queue full of tokens if (Zookeeper.exists (Startpath, false)! = NULL) {
                Zookeeper.create (Startpath, NULL, ZooDefs.Ids.OPEN_ACL_UNSAFE, createmode.persistent);
                } final Countdownlatch latch = new Countdownlatch (1);
                        Final Watcher Previouslistener = new Watcher () {public void process (Watchedevent event) {
                        if (event.gettype () = = eventtype.nodedeleted) {latch.countdown ();

                }
                    }
                };
                If the node does not exist there will be an exception zookeeper.exists (Startpath, Previouslistener);
                Latch.await ();

            Offer (element); }
        }catch (Zknonodeexception e) {logger.error ("", e);
        } catch (Exception e) {throw exceptionutil.converttoruntimeexception (e);
    } return true; }/** * Fetch data from the queue, when there is a start flag bit, start fetching data, all the data is finished before deleting the start flag * * @return * @throws Exception * * Suppresswarnings ("unchecked") public T poll () throws Exception {try {//Queue not yet full if (Z
                Ookeeper.exists (Startpath, false) = = null) {final Countdownlatch latch = new Countdownlatch (1);
                        Final Watcher Previouslistener = new Watcher () {public void process (Watchedevent event) {
                        if (event.gettype () = = eventtype.nodecreated) {latch.countdown ();

                }
                    }
                };

                If the node does not exist there will be an exception zookeeper.exists (Startpath, Previouslistener);
         An exception occurs if the node does not exist       Latch.await ();
            } list<string> List = Zookeeper.getchildren (root, false);
            if (list.size () = = 0) {return null;  }//Sort the queue in order from small to large collections.sort (list, new comparator<string> () {public int compare (string lhs, string rhs) {return Getnodenumber (LHS, Node_name). CompareTo (Getnodenumber (RH
                S, Node_name));

            }
            });
                /** * Loops The elements in the queue and then constructs the complete path to read the data through this path */for (String nodename:list) {
                String Nodefullpath = Root.concat ("/"). Concat (NodeName);
                    try {T node = (t) bytetoobject (Zookeeper.getdata (Nodefullpath, FALSE, null));
                    Zookeeper.delete (Nodefullpath,-1);
                return node;
                } catch (Zknonodeexception e) {logger.error ("", e);
         }   } return null;
        } catch (Exception e) {throw exceptionutil.converttoruntimeexception (e); }}/** * method of intercepting the number of nodes * * @param str * @param nodeName * @return */Private St
        Ring Getnodenumber (String str, string nodeName) {int index = Str.lastindexof (nodeName);
            if (index >= 0) {index + = node_name.length (); return index <= str.length ()?
        Str.substring (Index): "";

    } return str; }

}

distributedqueuetest

Package zookeeper;  

Import java.io.Serializable; import org.  
I0Itec.zkclient.ZkClient; import org.  

I0Itec.zkclient.serialize.SerializableSerializer; public class Distributedqueuetest {public static void main (string[] args) {zkclient zkclient = new Zkclie  
       NT ("127.0.0.1:2181", the New Serializableserializer ());  
       distributedsimplequeue<sendobject> queue = new Distributedsimplequeue<sendobject> (zkClient, "/Queue");  
       New Thread (new Consumerthread (queue)). Start ();  

   New Thread (new Producerthread (queue)). Start ();  

   }} class Consumerthread implements Runnable {private distributedsimplequeue<sendobject> queue;  
   Public Consumerthread (distributedsimplequeue<sendobject> queue) {this.queue = queue; public void Run () {for (int i = 0; i < 10000; i++) {try {Thread.Sleep (int) (Math.random () * 5000));//random sleep a bit SendObjectSendObject = (sendObject) queue.poll ();  
           System.out.println ("Consume a message success:" + SendObject); } catch (Exception e) {}}}} class Producerthread implements Runnable {private D  

   Istributedsimplequeue<sendobject> queue;  
   Public Producerthread (distributedsimplequeue<sendobject> queue) {this.queue = queue; public void Run () {for (int i = 0; i < 10000; i++) {try {Thread.Sleep (int) (Math.random () * 5000));//random sleep a bit SendObject SendObject = new SendObject (string.valueof (i), "Conte  
               NT "+ i);  
               Queue.offer (SendObject);  
           SYSTEM.OUT.PRINTLN ("Send a message successfully:" + sendObject); } catch (Exception e) {}}}} class SendObject implements Serializable {private  

   Static final Long serialversionuid = 1L; Public SendObject (string ID, string content) {THIS.Id = ID;  
   this.content = content;  

   } private String ID;  

   Private String content;  
   Public String GetId () {return id;  
   } public void SetId (String id) {this.id = ID;  
   } public String GetContent () {return content;  
   The public void SetContent (String content) {this.content = content;  
   } @Override Public String toString () {return "SendObject [id=" + ID + ", content=" + content + "]";   }  

}

Distributedsimplequeue

Package zookeeper;  
Import java.util.Collections;  
Import Java.util.Comparator;  
Import java.util.List; import org.  
I0Itec.zkclient.ExceptionUtil; import org.  
I0Itec.zkclient.ZkClient; import org.  
I0Itec.zkclient.exception.ZkNoNodeException;  
Import Org.slf4j.Logger;  


Import Org.slf4j.LoggerFactory; /** * Distributed queue, producer, consumer implementation * @author Linbingwen * * @param <T> * * public class DISTRIBUTEDSIMPLEQUEUE&LT;T&G T  

    {private static Logger Logger = Loggerfactory.getlogger (Distributedsimplequeue.class); Protected final zkclient zkclient;//for Operation zookeeper cluster protected final String root;//stands for root node protected static FINA L String node_name = "n_";//the name of the sequential node public distributedsimplequeue (Zkclient zkclient, String root) {thi  
        S.zkclient = zkclient;  
    This.root = root; }//Gets the size of the queue public int size () {/** * by getting the list of child nodes under the root node */return Zkclie  
    Nt.getchildren (Root). Size ();  }

    Determines whether the queue is empty public boolean isEmpty () {return Zkclient.getchildren (root). Size () = = 0; /** * Provides data to the queue * @param element * @return * @throws Exception */Public Boolea n Offer (T element) throws exception{//The full path of the build data node is String Nodefullpath = root. Concat ("/"). Concat (  
        Node_name);  
        try {//Create persistent node, write Data zkclient.createpersistentsequential (Nodefullpath, Element);  
            }catch (zknonodeexception e) {zkclient.createpersistent (root);  
        Offer (element);  
        } catch (Exception e) {throw exceptionutil.converttoruntimeexception (e);  
    } return true;  

            }//Fetch data from queue @SuppressWarnings ("unchecked") public T poll () throws Exception {try {  
            list<string> list = Zkclient.getchildren (root);  if (list.size () = = 0) {return null;
            }//Sort the queue in order from small to large collections.sort (list, new comparator<string> () { public int Compare (string lhs, string rhs) {return Getnodenumber (LHS, node_name). compar  
                ETo (Getnodenumber (RHS, node_name));  

            }  
            });  

                /** * Loops The elements in the queue and then constructs the complete path to read the data through this path */for (String nodename:list) {      
                String Nodefullpath = Root.concat ("/"). Concat (NodeName);  
                    try {t node = (t) zkclient.readdata (Nodefullpath);  
                    Zkclient.delete (Nodefullpath);  
                return node;  
                } catch (Zknonodeexception e) {logger.error ("", e);  

        }} return null;  
        } catch (Exception e) {throw exceptionutil.converttoruntimeexception (e); }} PrivatE string Getnodenumber (String str, string nodeName) {int index = Str.lastindexof (nodeName);  
            if (index >= 0) {index + = node_name.length (); return index <= str.length ()?  
        Str.substring (Index): "";  

    } return str;   }  

}

Run Result:

Send a message success: SendObject [Id=0, content=content0] consume a message success: SendObject [Id=0, content=content0] consume a message success: null Send a message success: SendObject [Id=1, Content=content1] consume a message success: SendObject [Id=1, content=content1] Send a message successfully: SendObject [id=2,
CONTENT=CONTENT2] Consume a message success: SendObject [id=2, Content=content2] Send a message success: SendObject [id=3, Content=content3] Send a message success: SendObject [id=4, CONTENT=CONTENT4] consume a message success: SendObject [id=3, Content=content3] consume a message success: SendObject [Id=4,
CONTENT=CONTENT4] Send a message success: SendObject [id=5, CONTENT=CONTENT5] consume a message success: SendObject [id=5, CONTENT=CONTENT5]
Consume a message success: null sends a message successfully: SendObject [id=6, CONTENT=CONTENT6] Send a message successfully: SendObject [Id=7, content=content7] Consume a message success: SendObject [id=6, CONTENT=CONTENT6] Send a message success: SendObject [Id=8, content=content8] consume a message success: SendObject [Id=7,
CONTENT=CONTENT7] Consume a message success: SendObject [Id=8, content=content8] Send a message success: SendObject [Id=9, Content=content9] Consume a message success: SendObject [Id=9, content=content9] consume a message success: null sends a message success: SendObject [id=10, content=content10] consume a message success: SeNdobject [id=10, content=content10]
 

Code Source:
https://blog.csdn.net/evankaka/article/details/70806752