High-performance Server analytics-Rpcserver of Hadoop

First, Listener

The listener thread, when the server is running, is responsible for listening for connections from the client and using select mode to handle the Accept event.

At the same time, it turns on an idle connection (idle Connection) processing routine, which is closed if there is an expired idle connection. This routine is implemented by a timer.

When a select operation is called, it may block, which gives the opportunity for other threads to execute. When an accept event occurs, it is awakened to handle the entire event, and processing the event is a doaccept call.

Doaccept:

voidthrows InterruptedException, IOException,  OutOfMemoryError {      ServerSocketChannel server = (ServerSocketChannel) key.channel();      SocketChannel channel;      whilenull) {        channel.configureBlocking(false);        channel.socket().setTcpNoDelay(tcpNoDelay);        channel.socket().setKeepAlive(true);        Reader reader = getReader();        Connection c = connectionManager.register(channel);        key.attach(c);  // so closeCurrentConnection can get the object        reader.addConnection(c);      }    }

While multiple connections may initiate a request at the same time, a while loop is used here.

The key here is to set up a new socket for non-blocking, which is based on performance considerations, non-blocking ways to read the data in the socket receive buffer as much as possible, This guarantees that the future will call this socket to receive the reader and send the responder thread will not be blocked because of the send and receive, if the entire communication process is busy, then reader and responder thread can try to not block on I/O, This can reduce the number of thread context switches and increase CPU utilization.

Finally, a reader is acquired to join this connection to the reader's buffer queue, while the Connection Manager monitors and manages the lifetime of the connection.

Here's how to get reader:

    //最简单的负载均衡    Reader getReader() {      1) % readers.length;      return readers[currentReader];    }

Second, Reader

When a newly established connection is added to reader's buffer queue pendingconnections, reader is also awakened to handle data reception on this connection.

      publicvoidaddConnectionthrows InterruptedException {        pendingConnections.put(conn);        readSelector.wakeup();      }

Multiple reader threads are configured in the server, apparently to improve the ability of concurrent service connections.

Here's the main logic for reader:

while(true) {        ...       //取出一个连接，可能阻塞       Connection conn = pendingConnections.take();       //向select注册一个读事件       conn.channel.register(readSelector, SelectionKey.OP_READ, conn);       ...       //进行select，可能阻塞       readSelector.select();       ...       //依次读取数据       for(keys){            doRead(key);       }       ...}

When the server is still running, the reader thread handles as many connections as possible in the buffer queue, registers each connected read event, and uses select mode to get notifications of data received on the connection. When there is data that needs to be received, it does its best to read the data on the connection returned by select to prevent the listener thread from starving (starving) because there is no running time.

If the listener thread is hungry, the result is a sharp drop in concurrency capability, and a new connection request from the client timed out or could not be established.

Note When you get a connection from a buffer queue, reader may block because it takes the take method in the Linkedblockingqueue class, which blocks when the queue is empty, so that the reader thread can block to the time that the other thread executes.

There are two wake-up time for reader threads:
1. Listener established a new connection and added this connection to the buffer queue of 1 reader;
2. The select call returns.

In the Doread call of reader, it calls the Readandprocess method, which loops through the data, receives the header of the packet, the context header, and the real data.
This process is worth mentioning in the following Channelread method:

  privateint channelRead(ReadableByteChannel channel,                           ByteBuffer buffer) throws IOException {    intcount = (buffer.remaining() <= NIO_BUFFER_LIMIT) ?                null, buffer);    if (count0) {      rpcMetrics.incrReceivedBytes(count);    }    returncount;  }

Channelread will judge the data received in the array BUFFER of the remaining unread data, if it is greater than a critical value nio_buffer_limit, take the technique of sharding to read multiple times to prevent the JDK to large BUFFER to take the change to direct Buffer optimization.

This, perhaps, takes into account that direct buffer has some overhead when it is built, and that direct buffer is not recycled by GC before jdk1.6 because they are allocated in the memory space outside the JVM's heap.

Exactly how the effect of this optimization, without testing, also skipped. Perhaps to reduce the burden on the GC.

After reader reads a full rpcrequest package, the Processonerpc method is called and the call goes into the business logic segment. This method, from the received packet, deserializes the rpcrequest header and data, constructs a Rpcrequest object, sets the tracing information required by the client (trace info), and constructs a call object, as shown in:

Thereafter, when handler is processed, it is in call, which is a encapsulated object that contains information related to the connection.

After you have the call object, add it to the server's Callqueue queue for handler processing. Because the method is used put , reader will block when the Callqueue is full (handler busy), as shown below:

callQueue.put(call);              // queue the call; maybe blocked here

Third, Handler

Handler is to invoke the appropriate business logic interface to process the request based on the method (call) and parameters in the RPC request.

There is more than one handler in a server that corresponds to multiple business interfaces, and this article does not discuss specific business logic.

The logic of handler is basically as follows (omitting exceptions and other minor information):

 Public void Run() {Server.set (SERVER. This); Bytearrayoutputstream buf =NewBytearrayoutputstream (initial_resp_buf_size); while(running) {Try{FinalCall call = Callqueue.take ();//Pop the queue; maybe blocked hereCurcall.set (call);Try{if(Call.connection.user = =NULL) {value = Call (Call.rpckind, Call.connection.protocolName, call.rpcrequest, call            . timestamp); }Else{value = Call.connection.user.doAs (...); }          }Catch(Throwable e) {//slightly ...} curcall.set (NULL);synchronized(call.connection.responseQueue) {responder.dorespond (call); }  }

Visible, Handler takes a call from the Callqueue and calls the Server.call method, and finally calls the responder Doresponde method to send the result to the client.

Server.call Method:

    publiccall(RPC.RpcKind rpcKind, String protocol,        longthrows Exception {      return getRpcInvoker(rpcKind).call(this, protocol, rpcRequest,          receiveTime);    }

Iv. Responder

A server has only 1 responder threads.

This thread continually makes several important calls to reconcile and send data with handler:

//这个wait是同步作用，具体见下面分析waitPending();     ...//开始select，或许会阻塞writeSelector.select(PURGE_INTERVAL);...//如果selectKeys有数据，就依次异步发送数据for(selectorKeys){    doAsyncWrite(key);}...//当到达丢弃时间，会从selectedKeys构造calls，并依次丢弃for(Call call : calls) {  doPurge(call, now);}

When handler calls the Dorespond method, the results of handler processing are added to the ResponseQueue's tail instead of being sent back to the client immediately:

    voidthrows IOException {      synchronized (call.connection.responseQueue) {        call.connection.responseQueue.addLast(call);        if1) {          //注意这里isHandler = true,表示可能会向select注册写事件以在Responder主循环中通过select处理数据发送          true);        }      }    }

The above synchronized can be seen, responsequeue is contention resources, corresponding:

Handler is the producer, and the result is added to the queue;
Responder is the consumer who pulls the results out of the queue and sends them.

ProcessResponse initiates the responder for sending, first removing a call from ResponseQueue in a non-blocking manner, and then trying to send the call.rpcresponse in a non-blocking manner, and returns if the message is complete.

When the remaining data is not sent, call is inserted into the first position of the queue, because the Ishandler parameter is passed in as true in the calls from handler, so the Writeselector is awakened and a write event is registered, where the incpending () method, is to block responder threads when registering write events with selector, followed by analysis.

Call.connection.responseQueue.addFirst (call);if(Inhandler) {//Set the serve time when the response have to be sent laterCall.timestamp = Time.now (); Incpending ();Try{//Wakeup the thread blocked on select, only then can the call                //To Channel.register () complete.Writeselector.wakeup ();              Channel.register (Writeselector, selectionkey.op_write, call); }Catch(Closedchannelexception e) {//its OK. Channel might be closed else where.Done =true; }finally{decpending (); }            }

Go back to the main loop of responder and see what happens if you register a write event to select:

          when executing this sentence, if the handler called Responder.doresonde () is registering a write event with the Select, it will block          //Purpose It is clear that for the next sentence the select can fetch data and return immediately, which reduces the number of blocking occurrencesWaitpending ();//If A channel is being registered, wait.          //Here with timeout blocking to select, is to be able to periodically wake up when no data is sent to handle long-standing callWriteselector.select (Purge_interval); Iterator<selectionkey> iter = Writeselector.selectedkeys (). Iterator (); while(Iter.hasnext ())            {Selectionkey key = Iter.next (); Iter.remove ();Try{if(Key.isvalid () && key.iswritable ()) {//Asynchronous SendDoasyncwrite (key); }            }Catch(IOException e) {Log.info (Thread.CurrentThread (). GetName () +": Doasyncwrite threw Exception"+ e); }          }

The key content has been commented, no longer repeat. As you can see, considering both synchronization and performance, this is a worthwhile place to learn.

V. Summary

This article focuses on the server portion of the RPC call in Hadoop, as you can see, this is a sophisticated design, considered very thin.

2. About synchronization:
   Listener production, reader consumption, reader production, handler consumption, handler production, responder consumption.
   Therefore, they must be synchronized, Hadoop implementation, both using Blockingqueue put&take operation to achieve blocking, for synchronization purposes, but also for contention resources using synchronized to achieve synchronization.

4. About Buffering:
   Several of these buffer queues are also noteworthy because the server has a particularly long concurrent request, and handler is slow when executing call for business logic, so buffering between requests and processing must be established.
   Also, there is a rate mismatch between processing and sending, which also requires buffering.

6. About the threading model:
   Listener single-threaded, reader multi-threaded, Handler multi-threaded, responder single-threaded, why is this design?

   Listener uses select mode to handle the Accept event, a client in a period of time generally only establish a limited number of connections, and the establishment of the connection is relatively fast, so the single-threaded enough to deal with, after the establishment of the direct throw to reader, so that they can easily cope with the new connection.

   Handler multi-threading, business logic is big head, it may be involved in computational dense or I/O intensive, if the thread is small, too long business logic may cause most of the handler threads to block, so the brisk business logic must be queued and not be processed in time, If reader collects the request queue for a long time, the whole communication must deteriorate, so this is a typical time to need concurrency, the context of this moment of the switch is necessary, not tangled, concurrency.

   Responder is single-threaded, obviously, responder will be more relaxed, because although the request is many, but after the Reader->handler buffer and handler processing time, the last batch can send the result has been sent. Responder more is to collect and process those long results, and to get results and send them through the efficient select mode.
   Over here Handler calls the Responder.dorespond send directly after the business logic call, because this is a call to return immediately, this call is time-consuming, so it is not necessary for handler to be blocked because of the send, further fully handler multi-threading ability, reduce the chance of thread switching , play its multi-threaded concurrency advantages, but also for responder burden, in order to enhance responder single-threaded combat confidence.

8. About locks
   Because of the synchronization requirements, so locking is essential, synchronized way lock, can significantly reduce the complexity of synchronization between this complex object, reduce the occurrence of errors.

"Can be reproduced, annotated source"
"Copyright @foreach_break Blog Address http://blog.csdn.net/gsky1986"

High-performance Server analytics-Rpcserver of Hadoop