RPC Communication function implementation

Table of Contents

RPC Communication function implementation

Configuration parameters

Calling methods

RPC Communication function implementation

The RPC communication function of HBase is mainly based on the two components of Protobuf and NiO, and the Protobuf Blockingrpcchannel (blocking) is selected on the communication pipeline. The Callblockingmethod method determines the interaction between client and server, such as the method used to communicate and the format rules of communication messages are described by this method.

HBase declares that the Blockingrpcchannelimplementation implementation class is used to implement the business logic of the Blockingrpcchannel interface, which uses socket communication in the choice of communication mode. In the service side of the communication to build ServerSocket through Rpcserver, while the client uses Rpcclient to build sockets for communication with the server, according to functional responsibilities, rpcserver can be divided into 3 large components, including:

• Listener is responsible for listening to client connection requests

  Within the listener, a serversocketchannel and multiple reader threads are primarily encapsulated, where Serversocketchannel is primarily responsible for receiving client connection requests, which are staged in the wait queue before the request is responded to. The length of the wait queue is set by the Hbase.ipc.server.listen.queue.size parameter (default is 128). For each established connection, the system also detects its idle time in real time, and if the idle time exceeds 2 seconds (that is, the client does not send the request again through the connection in 2 seconds, and the previous request operation has been processed), the system shuts down the connection. The time threshold is controlled by the Hbase.ipc.client.connection.maxidletime parameter.

  When the requested information arrives, it dispatches the appropriate reader to read (the load on each reader is balanced based on rotation), The number of reader threads is specified by the hbase.ipc.server.read.threadpool.size parameter, which defaults to 10, and the thread starts up into a blocking state until the client requests the operation. The communication messages sent by the client to the server are organized in a certain format:

  
  
  2. When the client makes the initial handshake with the server, it sends the RPCHEADER message to the server so that the server can verify the client's connection request.

     If the validation result satisfies the following rules, the request operation is legal:

     (1) The first 4 bytes of information is HBAs;

     (2) The value of the 5th byte (version information) is 0;

     (3) In the case where security is not enabled (the Hbase.security.authentication property value is not Kerberos), the value of the 6th byte is 80.

  4. Next, the client sends a CONNECTIONHEADER message to the server to encapsulate the service requested by the client. The

     Connectionheader is serialized by using PROTOBUF, with its Protocol declaration as follows:

     message    Connectionheader {Optional UserInformation user_info = 1;    Optional String service_name = 2;    Optional String cell_block_codec_class = 3; Optional String cell_block_compressor_class = 4;} 

     After the

     server receives the request message, it can use its Service_Name property to determine the name of the service to be accessed by the client, thereby locating the specific service.

  6. Once the specific service is identified, the client can continuously send a request message to the server to locate which method of the service will be executed.

     The method name is encapsulated by Requestheader, which is part of the request message:

     
     

     The Requestheader is also serialized with Protobuf, whose protocol declaration is as follows:

     Message Requestheader {    optional UInt32 call_id = 1;    Optional Rpctinfo trace_info = 2;    Optional String method_name = 3;    Optional BOOL Request_param = 4;    Optional Cellblockmeta Cell_block_meta = 5;    Optional UInt32 priority = 6;}

     The Method_name property is used to locate the name of the method to be executed, the method parameter is encapsulated by the Param message, and in addition, the client can pass some keyvalue data to the server (such as the replication function will use this data). The data is serialized into the cellblock message. After the reader thread reads this information, it starts constructing the Callrunner object and assigns it to the idle handler for processing.

• Handler is responsible for handling the client's request operation

  From a server-side perspective, all client requests can be encapsulated into Callrunner objects, and if reader is seen as a producer of Callrunner, then handler is the consumer. To speed up the responsiveness of the service side, Rpcserver is allowing multiple consumers to concurrently consume all Callrunner products. But how should the Callrunner product be distributed among all consumers? This is mainly dispatched by Rpcscheduler. HBase declares two types of Rpcscheduler function implementations, where Hmaster uses Fiforpcscheduler and Hregionserver uses Simplerpcscheduler.

  2. Fiforpcscheduler

     Based on the line pool consumption of all the Callrunner products, callrunner consumption sequence using FIFO principle (according to the Order of output), for each Callrunner product, the system will open a handler thread responsible for its consumption processing, The maximum number of concurrent threads that the thread pool can allow is declared externally by a specific service, such as Hmaster, which allows 25 concurrent handler (set by Hbase.master.handler.count parameters) by default.

  3. Simplerpcscheduler

     After scheduling with this policy, all Callrunner products are divided into 3 groups according to the different request types:

     (1) If the request of its encapsulation is based on the operation of the meta-table, it is divided into the Priorityexecutor group;

     (2) If the request of its encapsulation is based on the operation of the user table, it is divided into the Callexecutor group;

     (3) If it encapsulates a replication request, divide it into the Replicationexecutor group.

     Each product group is then assigned an unequal number of handler, allowing handler to consume only the products in the specified group. The number of handler assigned by different product groups is also declared by a specific service, taking Hregionserver for example:

     The number of handler assigned to the Priorityexecutor group is specified by the Hbase.regionserver.metahandler.count parameter, which defaults to 10;

     The number of handler assigned to the Callexecutor group is specified by the Hbase.regionserver.handler.count parameter, which defaults to 30;

     The number of handler assigned to the Replicationexecutor group is specified by the Hbase.regionserver.replication.handler.count parameter, which defaults to 3.

     Each product group can also be subdivided into multiple product queues, by default each product group contains only one product queue. All handler in the product group compete for the resources in that queue, and in order to prevent a fierce competition, each product group can be divided into multiple product queues, allowing each handler to preempt only the resources in the specified queue. In Hregionserver, you can calculate how many product queues a callexecutor group can be divided into by the following methods:

     Math.max (1,hbase.regionserver.handler.count*hbase.ipc.server.callqueue.handler.factor)

     Where the Hbase.ipc.server.callqueue.handler.factor property value defaults to 0, that is, by default, only the product group is divided into one product queue.

     The capacity of a single product queue is not infinitely growing on demand, and HBase has a corresponding threshold control over its length and space size, where:

     Hbase.ipc.server.max.callqueue.length is used to limit the length of the product queue (default is handler number multiplied by 10)

     Hbase.ipc.server.max.callqueue.size is used to limit the amount of space in the product queue (default is 1G)

     After the Callrunner product is successfully assigned to handler, the handler begins to consume it, and the consumption process is mainly done by invoking the Rpcserver call method to execute the corresponding method of the specified service. The results of the execution of the method are returned to the client through responder.

  5. Responder is responsible for returning processing results from the server to the client

     The communication messages returned to the client by the server are organized in the following format:

     
     

     Where Responseheader is serialized using PROTOBUF, its Protocol declaration is as follows:

     Message Responseheader {    optional UInt32 call_id = 1;    Optional Exceptionresponse exception = 2;    Optional Cellblockmeta Cell_block_meta = 3;}

     Its internal mainly encapsulates the service side of the execution exception information, as well as cellblock metadata information; result is used to encapsulate the return result of the execution method, whose serialization method needs to be determined according to the specific return value type; Cellblock is used to encapsulate keyvalue data returned by the server (such as the query results for a scan operation).

After the client sends the request message, it enters the loop wait state until the server returns the execution result, and if the wait time exceeds 10 seconds, the system considers the request to be unsuccessful and will turn on retry or close the connection ( If the hbase.ipc.client.connect.max.retries parameter value is 0).

Configuration parameters

• Server-side related configurations are as follows:

  1. hbase.ipc.server.listen.queue.size

     The waiting queue length for the connection request, which defaults to the same as the Ipc.server.listen.queue.size parameter value, is 128.

  2. Hbase.ipc.server.tcpnodelay

     Whether the Nagle algorithm is enabled during TCP communication and is not enabled by default.

  3. hbase.ipc.server.tcpkeepalive

     Whether to enable TCP keepalive mechanism, through the heartbeat packet to determine whether the connection is broken, enabled by default.

  4. hbase.ipc.server.read.threadpool.size

     Number of reader threads, default is 10.

  5. hbase.ipc.server.max.callqueue.size

     The maximum amount of storage space allowed for a single consumption queue (default is 1GB). Exceeding this limit the client throws the following exception:

     Call queue was full, is ipc.server.max.callqueue.size too small?

  6. hbase.ipc.server.max.callqueue.length

     The length limit of a single consumption queue, the default value is 10 times times the number of handler.

  7. Hbase.ipc.server.callqueue.handler.factor

     This parameter is used to determine the number of consumption queues.

  8. Hbase.ipc.server.callqueue.read.share

     The ratio of read handler to total handler count.

• The client-related configuration is as follows:

  2. Hbase.ipc.ping.interval

     The heartbeat interval between the client and the server, and the default read/write timeout for the socket (some other parameters of HBase override this value, such as Hbase.rpc.timeout).

  4. Hbase.client.rpc.codec

     Cellblock the encoding/decoder of the message content, the default is the same as the Hbase.client.default.rpc.codec parameter value, which is org.apache.hadoop.hbase.codec.KeyValueCodec.

     If the HBASE.CLIENT.DEFAULT.RPC.CODEC is set to an empty string and the Hbase.client.rpc.codec parameter is not set, the keyvalue will not be serialized in the RPC communication process using the Cellblock message, but the sequence into PROTOBUF's message (param or result).

  6. Hbase.client.rpc.compressor

     Compression/decompression algorithm for Cellblock message content, not compression by default.

  8. Hbase.ipc.socket.timeout

     The client tries to establish a connection time-out with the server, which defaults to ipc.socket.timeout for 20 seconds.

  10. Hbase.rpc.timeout

      The client's RPC request to regionserver time-out.

  12. Hbase.client.pause

      After the socket connection fails, it sleeps for a period of time and then re-connects, which is used to specify how long to hibernate and the default is 0.1 seconds.

  14. Hbase.ipc.client.connect.max.retries

      When there is an error in the connection between the client and the server, this parameter specifies the number of retries and defaults to 0 (no retries).

  16. Hbase.ipc.client.connection.maxidletime

      When the client and server connection idle time exceeds this value (that is, the client does not receive response information from the server for a specified time range), the connection is closed and the default response time-out is 10 seconds.

  18. Hadoop.rpc.socket.factory.class.default

      The Socketfactory implementation class, which defaults to Org.apache.hadoop.net.StandardSocketFactory, Createsocket method creates Socketchannel for NIO communication.

Calling methods

1. The client can build the required service with the following code, taking ClientService example:

   Rpcclient rpcclient = new Rpcclient (conf, Clusterid); Blockingrpcchannel channel =     Rpcclient.createblockingrpcchannel (serverName, user, rpctimeout); Clientservice.blockinginterface stub = Clientservice.newblockingstub (channel);

   	First, build the Rpcclient, where the Clusterid attribute can be read from the/hbase/hbaseid node of the zookeeper;
   	The servername is used to locate the server-side address, which can be built through the servername.valueof ("${host},${port},${startcode}") method, such as servername.valueof ("localhost, 60020,1422961250317 ");
   	User is the client's request, which can be obtained by user.getcurrent ();
   	Rpctimeout the time-out for the RPC request.

2. The service can be published by the following code to the service, also take clientservice example:

   list<blockingserviceandinterface> services = new arraylist<    Blockingserviceandinterface> (); Services.add (New Blockingserviceandinterface ( Clientservice.newreflectiveblockingservice (Regionserver), ClientService.BlockingInterface.class)); Rpcserver rpcserver = new Rpcserver (serverinstance, Name, Services, ISA, Conf, scheduler); Rpcserver.start (); 

   	construct ClientService Real example, through its Newreflectiveblockingservice method, the method parameter is Hregionserver instance, it implements the Clientservice.blockinginterface interface;
   	ServerInstance as a service process instance , this is hregionserver;
   	Name is the service process name;
   	Services is a list of service included in the service process;
   	Isa is the communication address of the service;
   	Scheduler for RPC Request Scheduler, there are currently two implementations: Fiforpcscheduler and Simplerpcscheduler.

Copyright NOTICE: This article for Bo Master original article, without Bo Master permission not reproduced.

RPC Communication function implementation