Zookeeper Source Learning Notes (1)--client End parsing

Preface

Zookeeper is a relatively simple distributed coordination service, we can further understand the principle of distributed by reading the source code. Environment

ZooKeeper 3.4.9 Entry function

In bin/zkcli.sh, we see that the real portal of the client is actually a Org.apache.zookeeper.ZooKeeperMain Java class

"$JAVA" "-dzookeeper.log.dir=${zoo_log_dir}" "-dzookeeper.root.logger=${zoo_log4j_prop}" \
     -cp "$CLASSPATH" $ Client_jvmflags $JVMFLAGS \
     org.apache.zookeeper.ZooKeeperMain "$@"

Through the source of the daytime, see in the Zookeepermain is mainly composed of two parts

CONNECTTOZK (cl.getoption ("Server"));

while (line = (String) readline.invoke (console, Getprompt ()))! = null) {
  executeline (line);
}

Constructs a zookeeper object that establishes a communication connection with Zookeeperserver to invoke jline through reflection. The Consolereader class, which reads the terminal input and then invokes the zookeeper interface by parsing a single-line command.

As mentioned above, the client is actually a simple encapsulation of zookeeper.jar, and after constructing a zookeeper object, it calls the Zookeeper interface and the Server to interact by parsing the user input. ZooKeeper class

Just now we saw that the client-side interaction with ZooKeeper Server was actually done through the ZooKeeper object, and then we went into the ZooKeeper class in detail and looked at its interaction logic with the service side.

Public ZooKeeper (String connectstring, int sessiontimeout, Watcher Watcher,
            Boolean canbereadonly)
        throws IOException 
{
  Connectstringparser connectstringparser = new Connectstringparser (connectstring);
  Hostprovider Hostprovider = new Statichostprovider (connectstringparser.getserveraddresses ());
  CNXN = new Clientcnxn (Connectstringparser.getchrootpath (),
  Hostprovider, Sessiontimeout, this, Watchmanager, Getclientcnxnsocket (), canbereadonly);
  Cnxn.start ();
}

In ZooKeeper's construction method, you can see that the server address in ZooKeeper constructs a CLIENTCNXN class in which the system creates two new threads

Sendthread = new Sendthread (clientcnxnsocket);
Eventthread = new Eventthread ();

Where Sendthread is responsible for encapsulating zookeeper request information into a packet, sending it to the server, and maintaining the heartbeat of the same server, Eventthread is responsible for parsing Sendthread obtained through response, Then send to watcher::p Rocessevent for detailed event handling.

As shown above, in the client after the terminal Input command, will be encapsulated as a request requests, through submitrequest, further encapsulated into the packet package, submitted to Sendthread processing.

Sendthread sends the packet to the server through Dotransport, and obtains the result through readresponse, resolves to an event, and waits for execution in the queue where the event is added to the eventthread.

Eventthread through processevent the event events in the consumption queue. Sendthread

Sendthread's primary role is to maintain the heartbeat between the client and server in addition to packet packets to the server, ensuring that the session survives.

Now let's start from the source and see how Sendthread works.

Sendthread is a thread class, so let's go into its run () method and look at his startup process.

while (State.isalive ()) {
  if (!clientcnxnsocket.isconnected ()) {
    //start and server's socket link
    startconnect ();
  }
  //Based on last connection time, determine if timeout
  if (state.isconnected ()) {To
    = Readtimeout-clientcnxnsocket.getidlerecv ();
  } else {to
    = Connecttimeout-clientcnxnsocket.getidlerecv ();
  }
  if (to <= 0) {
    throw new sessiontimeoutexception (Warninfo);
  }
  Send Heartbeat Package
  if (state.isconnected ()) {
    if (timetonextping <= 0 | | clientcnxnsocket.getidlesend () > Max_send _ping_interval) {
      sendping ();
      Clientcnxnsocket.updatelastsend ();
    }
  }
  Send instruction information to the Server
  Clientcnxnsocket.dotransport (To, Pendingqueue, Outgoingqueue, clientcnxn.this);
}

From the code above, you can see that the main tasks of Sendthread are as follows:
1. Create a socket link between the same Server
2. Determine if the link timed out
3. Regularly send heartbeat tasks
4. Send the Zookeeper directive to a long link between server and server

Zookeeper constructs a Clientcnxnsocket object by acquiring the Zookeeper_client_cnxn_socket variable, which by default is the Clientcnxnsocketnio class

String clientcnxnsocketname = System
                . GetProperty (zookeeper_client_cnxn_socket);
if (Clientcnxnsocketname = = null) {
  clientcnxnsocketname = ClientCnxnSocketNIO.class.getName ();
}

In Clientcnxnsocketnio::connect we can see that there is a socket link created between the server and the servers.

Socketchannel sock = Createsock ();
Registerandconnect (sock, addr);

Timeout and Heartbeat

In Sendthread::run, you can see whether there are readtimeout and connettimeout two time-outs for links, and if the link times out, throw an exception and terminate sendthread.

In the absence of a timeout, the heartbeat data is sent again to avoid the access timeout if the last heartbeat time is determined to exceed the 1/2 time-out. Send ZooKeeper Instructions

In the time series diagram, we see that after entering the instruction from the terminal, we will parse it into a packet packet and wait for Sendthread to send it.

Taking Zookeeper::create as an example

Requestheader h = new Requestheader ();
H.settype (ZooDefs.OpCode.create);
Createrequest request = new Createrequest ();
Createresponse response = new Createresponse ();
Request.setdata (data);
Request.setflags (Createmode.toflag ());
Request.setpath (Serverpath);
if (ACL! = null && acl.size () = = 0) {
    throw new keeperexception.invalidaclexception ();
}
Request.setacl (ACL);
Replyheader r = cnxn.submitrequest (h, request, response, NULL);

Here the Create command, encapsulated as a createrequest, was turned into a packet package by submitrequest

Public Replyheader submitrequest (Requestheader H, record request, record response, Watchregistration Watchregi
    Stration) throws Interruptedexception {Replyheader r = new Replyheader ();
    Packet Packet = Queuepacket (h, R, request, response, NULL, NULL, NULL, NULL, watchregistration);
        Synchronized (packet) {while (!packet.finished) {packet.wait ();
}} return R; } Packet Queuepacket (Requestheader H, Replyheader R, record request, record response, AsyncCallback CB, Strin
    G Clientpath, String Serverpath, Object ctx, watchregistration watchregistration) {Packet Packet = null; Note that we don't generate the Xid for the packet yet. It is//generated later at Send-time, by a implementation of Clientcnxnsocket::d Oio (),//Where the packet is AC
    Tually sent.
        Synchronized (outgoingqueue) {packet = new packet (h, R, request, response, watchregistration); Packet.CB = CB;
        Packet.ctx = CTX;
        Packet.clientpath = Clientpath;
        Packet.serverpath = Serverpath;
        if (!state.isalive () | | closing) {conlosspacket (packet); } else {//If the client is asking to close the session then//mark as closing If (h
            . GetType () = = opcode.closesession) {closing = true;
        } outgoingqueue.add (packet);
    }} sendthread.getclientcnxnsocket (). WAKEUPCNXN ();
return packet; }

In Submitrequest, we further see that the request is encapsulated as a packet packet and added to the Sendthread::outgoingqueue queue for execution.

Note: Here we also see that the so-called synchronization method in the Zookeeper method is actually after the packet is committed to the sendthread, into a while loop, waiting for the process to be completed before jumping out of the procedure

In the Sendthread::run while loop, Zookeeper sends the packet packets in Outgoingqueue through Dotransport to the Server.

void Doio (list<packet> pendingqueue, linkedlist<packet> outgoingqueue, Clientcnxn cnxn) {
    if ( Sockkey.isreadable ()) {
        //Read response information
        sendthread.readresponse (incomingbuffer);
    }
    if (sockkey.iswritable ()) {
        Packet p = findsendablepacket (Outgoingqueue, Cnxn.sendThread.clientTunneledAuthenticationInProgress ());
        Sock.write (P.BB);
    }
}

Before DOIO sends the socket information, it gets the return data from the socket and is processed by Readresonse.

void Readresponse (Bytebuffer incomingbuffer) throws IOException {
     Bytebufferinputstream bbis = new Bytebufferinputstream (incomingbuffer);
     Binaryinputarchive Bbia = binaryinputarchive.getarchive (bbis);
     Replyheader REPLYHDR = new Replyheader ();
     Replyhdr.deserialize (Bbia, "header");
     if (replyhdr.getxid () = =-1) {
        Watcherevent event = new Watcherevent ();
        Event.deserialize (Bbia, "response");
        Watchedevent we = new Watchedevent (event);
        Eventthread.queueevent (We);
     }
}

In Readreponse, by parsing the data, we can get the Watchedevent object and press it into the Eventthread message queue for distribution Eventthread

public void Run () {
    while (true) {
        Object event = Waitingevents.take ();
        if (event = = Eventofdeath) {
            waskilled = true;
        } else {
            processevent (event);}
}

Consume the events in the queue by processevent in Eventthread and distribute them to different watcher watch events for registration and distribution

Typically in zookeeper, we add a watcher for the specified node to listen for node changes, taking Zookeeper:exist as an example

The watch contains the un-chroot path
watchregistration WCB = null;
if (Watcher! = null) {
    WCB = new Existswatchregistration (Watcher, Clientpath);
}

Final String Serverpath = Prependchroot (Clientpath);

Requestheader h = new Requestheader ();
H.settype (ZooDefs.OpCode.exists);
Existsrequest request = new Existsrequest ();
Request.setpath (Serverpath);
Request.setwatch (Watcher! = null);
Setdataresponse response = new Setdataresponse ();
Replyheader r = cnxn.submitrequest (h, request, response, WCB);

The approximate logic of the code is similar to create, but the wathcer is wrapped in a layer of existwatchregistration, and when the packet object completes the request, the Register method is called, Register watch to a different watch list depending on the watchregistration of the package and wait for the callback.

if (p.watchregistration! = null) {
    p.watchregistration.register (P.replyheader.geterr ());
}

In ZooKeeper, there are three types of watchregistration, respectively, corresponding to datawatchregistration,childwatchregistration,existwatchregistration. And in the Zkwatchmanager class according to each type of watchregistration, there is a map table is responsible for storage.

Private final map<string, set<watcher>> datawatches =
            new hashmap<string, set<watcher>> () ;
Private final map<string, set<watcher>> existwatches =
            new hashmap<string, set<watcher>> ( );
Private final map<string, set<watcher>> childwatches =
            new hashmap<string, set<watcher>> ( );

When Eventthread::p rocessevent, the corresponding watch list is obtained from three maps for message notification and processing based on the path to which the event belongs. Summary

The client side of the source analysis is over.

The source code of the

ZooKeeper Client is simple, with three separate threads handling the commands, distributing and responding to operations, and avoiding locking in multi-threaded operations as much as possible, on the basis of ensuring that each thread is independent of each other.