[Hadoop Source Code Reading] [6]-org. Apache. hadoop. ipc-ipc.client

1. Client running process

1) create a proxy object;

2) The proxy object calls the corresponding method (invoke ());

3) invoke calls the call method of the client object and sends a request (parameters and methods) to the server );

4) wait until the call method is completed;

5) return the request result.

 

2. Main Internal classes of the Client

The main classes are described as follows:

• 1. Call, indicating one RPC call request
• 2. Connection indicates the connection between a client and a server, and a connection is started by a thread.
• 3. connectionid: Connection identifier (including server address, protocol, and other connection configuration items)
• 4. parallelcall: requests for concurrent calls
• 5. parallelresults: execution result of the parallel call

 

3. Client call Process

3.0 an actual call

In dfsclient

   Return(Clientdatanodeprotocol) RPC. getproxy (clientdatanodeprotocol.Class, Clientdatanodeprotocol. versionid, ADDR, conf );

3.1 generate proxy

  Public     Static Versionedprotocol getproxy (Class  <?     Extends  Versionedprotocol  >  Protocol,  Long  Clientversion, inetsocketaddress ADDR, usergroupinformation ticket, configuration Conf, socketfactory factory,  Int  Rpctimeout)  Throws  Ioexception {...... Versionedprotocol proxy  = (Versionedprotocol) proxy. newproxyinstance (protocol. getclassloader (),  New  Class [] {Protocol },  New  Invoker (protocol, ADDR, ticket, Conf, factory, rpctimeout ));......  Return  Proxy ;} 

Invoker is a class that implements the invocationhandler interface.

 

3.2 The proxy object calls the corresponding method (invoke ())

Getproxy caller, use this proxy to call any protocol declared function, such as the dfsclient example in the previous example. If proxy. getblockinfo (…) is called (...); Will be converted into calling the invoker class's invoke Function

  PublicObject invoke (Object proxy, method, object [] ARGs)ThrowsThrowable {...... Objectwritable Value=(Objectwritable) client. Call (NewInvocation (method, argS), remoteid );......ReturnValue. Get ();}
 

Invocation is used to encapsulate method names and parameters as the data transmission layer. The key to remote calling is that invocation implements the writable interface. Invocation writes the called methodname to out in the write (dataoutput out) function, and writes the number of parameters of the called method to out, at the same time, the classname of the parameter is written out one by one, and all parameters are written out one by one. This determines that the parameters in the method called through RPC are either simple or string, either the class that implements the writable interface (the parameter knows how to serialize to stream), or the Array (the element of the array must also be of the simple type, string, class that implements the writable interface ).

Invocation serialization parameters are implemented using the following functions: org. Apache. hadoop. Io. objectwritable. writeobject

  Public     Void  Write (dataoutput out)  Throws  Ioexception {utf8.writestring (Out, methodname); Out. writeint (parameterclasses. Length ); For  (  Int  I  =     0  ; I  <  Parameterclasses. length; I  ++  ) {Objectwritable. writeobject (Out, parameters [I], parameterclasses [I], conf );}} 

 

3.3invoke calls the call method of the client object and sends a request (parameters and methods) to the server)

  Public  Writable call (writable Param, connectionid remoteid)  Throws  Interruptedexception, ioexception {call  =     New  Call (PARAM );  //  New Invocation (method, argS)    Connection connection  = Getconnection (remoteid, call );  //  Obtain the connection object. A client can have multiple connections.    Connection. sendparam (CALL );  //  Serialize the invocation (method, argS) function name and parameter to the server.      While  (  !  Call. Done) {call. Wait ();  //  Wait for the result the thread that calls the client is blocked here   }  Return  Call. value;  //  Return call results    } 

 

3.4 get the connection object getconnection

  Private  Connection getconnection (connectionid remoteid, call)  Throws Ioexception, interruptedexception {connection;  Do  {  Synchronized  (Connections) {connection  =  Connections. Get (remoteid );  If  (Connection  =     Null  ) {Connection  =    New  Connection (remoteid); connections. Put (remoteid, connection );}}}  While  (  !  Connection. addcall (CALL ));  //  It can be seen that a connection can have multiple call    Connection. setupiostreams ();  //  A new thread is created for each connection.      Return Connection ;} 

 

3.5 The connection thread waits for the acceptance result

  Public     Void  Run (){  Try  {  While  (Waitforwork ())  //  Timeout detection and other conditions connection close   {  //  Wait here for work-read or close connection    Receiveresponse () ;}} close ();}  Private     Void  Receiveresponse (){  Try  {  Int  ID  =  In. readint (); //  Try to read an ID    Call call  =  CILS. Get (ID );  Int  State  =  In. readint ();  //  Read call status      If  (State  = Status. Success. State) {writable Value  =  Reflectionutils. newinstance (valueclass, conf); value. readfields (in );  //  Deserialization of results    Call. setvalue (value );  //  Here    CILS. Remove (ID );}}  Catch  (Ioexception e) {markclosed (e );}} 

 

3.6 return results to notify the client thread

  Public     Synchronized     Void  Setvalue (writable value ){  This  . Value  =  Value; callcomplete ();}  Protected    Synchronized     Void  Callcomplete (){  This  . Done  =     True  ; Notify ();  //  Notify caller    } 

 

4.Asynchronous/Synchronization Model

The external interfaces of hadoop RPC are actually synchronous, but the internal implementation of RPC is actually an asynchronous message mechanism. Hadoop uses the thread wait/notify mechanism to Implement Asynchronous synchronization. After a call is sent, the wait request is processed and the response is received. the receiveresponse () method after notify, notify () method in call. setvalue. But now there is a problem. One connection has multiple calls. There may be multiple calls waiting to receive messages at the same time. After the client receives the response, how can we determine which request is the response? This is a hashtable <integer, call> In the connection. The integer is used to identify the call, so that the request and response can be matched.

 

5.Sequence Chart

 

6. Client class diagram

 

 

 

7.Reference

Http://weixiaolu.iteye.com/blog/1504898

Http://www.wikieno.com/2012/02/hadoop-ipc-client/

Http://caibinbupt.iteye.com/blog/280790

Http://blog.csdn.net/sxf_824/article/details/4842153

Http://blog.csdn.net/historyasamirror/article/details/6159248