Mina, Netty, twisted Together (ii): TCP message boundary problem and split message by row

The data may be transferred multiple times between the start of the TCP connection and the end of the connection, that is, the server and the client may transmit multiple messages to each other during the connection process. The ideal situation is that one message is sent to each party, and the other side immediately receives one, that is, a write corresponding to read at a time. However, reality does not always follow the script to go.

Mina Official document excerpt:

TCP guarantess delivery of all packets in the correct order. But there was no guarantee that one write operation on the sender-side would result in one read event on the receiving side. One Call of Iosession.write (Object message) by the sender can result in multiple messagereceived (iosession session, OBJEC T message) events on the receiver; and multiple calls of Iosession.write (Object message) can leads to a single messagereceived event.

Netty Official Document Excerpt:

In a stream-based transport such as TCP/IP, received data is stored into a socket receive buffer. Unfortunately, the buffer of a stream-based transport is not a queue of packets but a queue of bytes. It means, even if you sent, and messages as, independent packets, an operating system would not treat them as both message s but as just a bunch of bytes. Therefore, there is no guarantee so what are you read are exactly what your remote peer wrote.

The above two paragraphs express the same meaning: TCP is a byte-stream-based protocol, it can only guarantee that one party sends and the other party receives the data in the same order of bytes, but there is no guarantee that one party each send a message, the other party can complete receive a message. It is possible to send two others to merge them into one, and it is possible to send an opponent to split them into two pieces. So the demo in the last blog post can be a false demonstration. However, the server and the client on the same machine or LAN and other network speed is very good, this problem is still difficult to test out.

Give a simple example (this example comes from the official Netty document):

The message sender sent three strings:

However, the receiver may receive this:

Then the problem is very serious, the receiver can not separate the three messages, it will not be able to parse.

For this, Mina's official documentation provides the following solutions:

1. Use fixed length messages

Use a fixed-length message. For example, each length of 4 bytes, then receive the time by each 4 bytes split on it.

2. Use a fixed length header that indicates the length of the body

Use a fixed-length header,header to specify the length of the body (in bytes) to place the contents of the information in the body. For example, the body length specified in the header is 100 bytes, then 100 bytes after the header is the body, that is, the content of the information, 100 bytes of body behind the next message header.

3, using a delimiter; For example many text-based protocols append a newline (or CR LF pair) after every message

Use the delimiter character. For example, many text-content protocols add a newline character (CR LF, or "\ r \ n") after each message, which is a single message. Of course, you can also use other special symbols as delimiters, such as commas, semicolons, and so on.

Of course, in addition to the above mentioned 3 kinds of programs, there are other options. Some protocols may also be used for multiple scenarios. For example, in the HTTP protocol, the header section uses the CR LF line to differentiate each header, and the header uses Content-length to specify the body byte number.

The following, respectively, with Mina, Netty, twisted with the relevant API implementation by the newline character Cr LF to split the message.

MINA:

Mina can use Protocolcodecfilter to process binary data that is sent and received, and how it is processed depends on protocolcodecfactory or Protocolencoder, Protocoldecoder, After processing in Iohandler the Messagereceived event function gets the message is no longer iobuffer, but the other type you want, can be a string, Java object. The CR LF split message can be implemented here using Textlinecodecfactory (an implementation class of Protocolcodecfactory).

 Public classTCPServer { Public Static voidMain (string[] args)throwsIOException {ioacceptor acceptor=NewNiosocketacceptor (); //Add a filter to receive and send content according to "\ r \ n" SplitAcceptor.getfilterchain (). AddLast ("codec",                   NewProtocolcodecfilter (NewTextlinecodecfactory (Charset.forname ("UTF-8"), "\ r \ n", "\ r \ n"))); Acceptor.sethandler (NewTcpserverhandle ()); Acceptor.bind (NewInetsocketaddress (8080)); }    }    classTcpserverhandleextendsIohandleradapter {@Override Public voidExceptioncaught (iosession session, Throwable Cause)throwsException {cause.printstacktrace (); }        //Receive new Data@Override Public voidMessagereceived (iosession session, Object message)throwsException {//receive the client's data, here is no longer the Iobuffer type, but the stringString line =(String) message; System.out.println ("Messagereceived:" +Line ); } @Override Public voidSessioncreated (iosession session)throwsException {System.out.println ("Sessioncreated"); } @Override Public voidSessionclosed (iosession session)throwsException {System.out.println ("Sessionclosed"); }  }  

Netty:

Netty are designed to be similar to Mina and need to be channelpipeline with some channelhandler to process the raw data. Here, the received data is split by row with Linebasedframedecoder, and the data is stringdecoder to a string by the byte code. Similarly, after the received data has been processed, in the Channelread event function, the MSG parameter is no longer bytebuf but a string.

 Public classTCPServer { Public Static voidMain (string[] args)throwsinterruptedexception {eventloopgroup bossgroup=NewNioeventloopgroup (); Eventloopgroup Workergroup=NewNioeventloopgroup (); Try{Serverbootstrap b=NewServerbootstrap (); B.group (Bossgroup, Workergroup). Channel (Nioserversocketchannel.class). Childhandler (NewChannelinitializer<socketchannel>() {@Override Public voidinitchannel (socketchannel ch)throwsException {Channelpipeline pipeline=Ch.pipeline (); //Linebasedframedecoder splitting messages by rowPipeline.addlast (NewLinebasedframedecoder (80)); //then press UTF-8 encoding to convert to stringPipeline.addlast (NewStringdecoder (charsetutil.utf_8)); Pipeline.addlast (NewTcpserverhandler ());              }                      }); Channelfuture F= B.bind (8080). sync ();          F.channel (). Closefuture (). sync (); } finally{workergroup.shutdowngracefully ();          Bossgroup.shutdowngracefully (); }      }    }    classTcpserverhandlerextendsChannelinboundhandleradapter {//Receive new Data@Override Public voidChannelread (Channelhandlercontext ctx, Object msg) {//msg after stringdecoder type is no longer bytebuf but stringString line =(String) msg; System.out.println ("Channelread:" +Line ); } @Override Public voidchannelactive (Channelhandlercontext ctx) {System.out.println ("Channelactive"); } @Override Public voidchannelinactive (Channelhandlercontext ctx) {System.out.println ("Channelinactive"); } @Override Public voidexceptioncaught (Channelhandlercontext ctx, throwable cause) {cause.printstacktrace ();      Ctx.close (); }  }  

Twisted:

The design of the twisted is not the same as the design of the above two, so the message segmentation is not quite the same. The class that handles the event Tcpserverhandle no longer inherits protocol, but inherits the subclass Lineonlyreceiver of protocol. The event method that receives the new data is no longer a datareceived, but a linereceived provided by the Lineonlyreceiver. See twisted source words can find lineonlyreceiver inside actually implement DataReceived, and then split it by line, there is a new row of data on the call linereceived.

#-*-coding:utf-8–*-   fromTwisted.protocols.basicImportLineonlyreceiver fromTwisted.internet.protocolImportFactory fromTwisted.internetImportreactorclassTcpserverhandle (lineonlyreceiver):#The new connection is established    defConnectionmade (self):Print 'Connectionmade'        #Connection Disconnect    defconnectionlost (self, Reason):Print 'Connectionlost'        #receive a new row of data    deflinereceived (self, data):Print 'linereceived:'+Data Factory=Factory () Factory.protocol=Tcpserverhandle reactor.listentcp (8080, Factory) Reactor.run ()

The following three servers are tested with a Java client:

 Public classTcpClient { Public Static voidMain (string[] args)throwsIOException {Socket Socket=NULL; OutputStream out=NULL; Try{Socket=NewSocket ("localhost", 8080); out=Socket.getoutputstream (); //Request ServerString lines = "Bed before the moon light \ r \ NAND is the ground frost \ r \ jutou wang \ \ \ \ \ \ \ \ \ \"; byte[] outputbytes = Lines.getbytes ("UTF-8");              Out.write (outputbytes);            Out.flush (); } finally {              //Close ConnectionOut.close ();          Socket.close (); }      }  }  

Mina Server output results:

sessioncreated
Messagereceived: Moon Light in front of the bed
Messagereceived: Suspicion is ground frost
Messagereceived: Jutou looking at the Moon
Messagereceived: Think of your hometown
Sessionclosed

Netty Server Output results:

Channelactive
Channelread: Moon Light in front of the bed
Channelread: Suspicion is ground frost
Channelread: Jutou looking at the Moon
Channelread: Think of your hometown
Channelinactive

Twisted Server Output results:

Connectionmade
Linereceived: Moon Light in front of the bed
Linereceived: Suspicion is ground frost
Linereceived: Jutou looking at the Moon
Linereceived: Think of your hometown
Connectionlost

Of course, the test can also be used to simulate the data sent to non-rule segmentation, the following with a more perverted client to test:

 Public classTcpClient { Public Static voidMain (string[] args)throwsIOException, interruptedexception {socket socket=NULL; OutputStream out=NULL; Try{Socket=NewSocket ("localhost", 8080); out=Socket.getoutputstream (); String Lines= "Front of Bed"; byte[] outputbytes = Lines.getbytes ("UTF-8");              Out.write (outputbytes);                            Out.flush (); Thread.Sleep (1000); Lines= "Moon"; Outputbytes= Lines.getbytes ("UTF-8");              Out.write (outputbytes);                            Out.flush (); Thread.Sleep (1000); Lines= "Light \ r \ n is ground frost \ \ Jutou"; Outputbytes= Lines.getbytes ("UTF-8");              Out.write (outputbytes);                            Out.flush (); Thread.Sleep (1000); Lines= "Looking at the Moon \ \ \ \ \ \ \"; Outputbytes= Lines.getbytes ("UTF-8");              Out.write (outputbytes);                        Out.flush (); } finally {              //Close ConnectionOut.close ();          Socket.close (); }      }  }  

Test the above three servers again, and the result is the same as the output above, without any problems.

Mina, Netty, twisted Together (ii): TCP message boundary problem and split message by row