Mina, Netty, and Twisted: TCP Message boundary and line-based message splitting. nettytwisted

Mina, Netty, and Twisted: TCP Message boundary and line-based message splitting. nettytwisted

Data may be transmitted multiple times from the beginning to the end of the TCP connection, that is, multiple messages may be transmitted between the server and the client during the connection process. Ideally, each Party sends a message, and the other party receives one message immediately, that is, one write corresponds to one read. However, the reality is not always based on the script.

Excerpt from the MINA official document:

TCP guarantess delivery of all packets in the correct order. but there is no guarantee that one write operation on the sender-side will result in one read event on the processing side. one call of IoSession. write (Object message) by the sender can result in multiple messageReceived (IoSession session, Object message) events on the receiver; and multiple cballs of IoSession. write (Object message) can lead to a single messageReceived event.

Excerpt from the Netty official document:

In a stream-based transport such as TCP/IP, stored ed data is stored into a socket receive buffer. unfortunately, the buffer of a stream-based transport is not a queue of packets but a queue of bytes. it means, even if you sent two messages as two independent packets, an operating system will not treat them as two messages but as just a bunch of bytes. therefore, there is no guarantee that what you read is exactly what your remote peer wrote.

The above two paragraphs share the same meaning: TCP is based on the byte stream protocol. It can only ensure that one Party sends the data in the same byte sequence as the other party receives the data. However, it is not guaranteed that each Party sends a message, and the other party can receive a complete message. It is possible that two parties are sent to merge them into one, or the other party may be sent to split them into two. So the Demo in the previous blog is a false demonstration. However, when the server and the client are on the same machine or have a good network speed, it is still difficult to test this problem.

Here is a simple example (this example is from the Netty official document ):

The message sender sends three strings:

However, what the recipient receives may be:

The problem is very serious. The receiver cannot separate the three pieces of information, so it cannot be parsed.

MINA's official documentation provides the following solutions:

1. use fixed length messages

Use a fixed-length message. For example, if each length is 4 bytes, it can be split by 4 bytes.

2. use a fixed length header that indicates the length of the body

Use a fixed-length Header to specify the length of the Body (number of bytes) and put the information content in the Body. For example, if the length of the Body specified in the Header is 100 bytes, the second byte after the Header is the Body, that is, the information content. The second byte Body is followed by the next Header.

3. using a delimiter; for example plain text-based protocols append a newline (or cr lf pair) after every message

Use separators. For example, many text content protocols add a line break (cr lf, that is, "\ r \ n") after each message, that is, one message line. Of course, you can also use other special characters as separators, such as commas and semicolons.

Of course, besides the three solutions mentioned above, there are other solutions. Some protocols may also use the above solutions. For example, in the HTTP protocol, the Header part uses cr lf line breaks to distinguish each Header, while in the Header, Content-Length is used to specify the number of Body bytes.

Next, we use the APIS provided by MINA, Netty, and Twisted to separate messages based on the linefeed cr lf.

MINA:

MINA can use ProtocolCodecFilter to process the binary data sent and received. The processing depends on ProtocolCodecFactory, ProtocolEncoder, and ProtocolDecoder, after processing, the message obtained by the messageReceived event function in IoHandler is no longer IoBuffer, but other types you want, such as strings and Java objects. Here, we can use TextLineCodecFactory (an implementation class of ProtocolCodecFactory) to implement cr lf message segmentation.

Public class TcpServer {public static void main (String [] args) throws IOException {IoAcceptor acceptor = new NioSocketAcceptor (); // Add a Filter, the content used for receiving and sending is divided into acceptor according to "\ r \ n. getFilterChain (). addLast ("codec", new ProtocolCodecFilter (new TextLineCodecFactory (Charset. forName ("UTF-8"), "\ r \ n", "\ r \ n"); acceptor. setHandler (new TcpServerHandle (); acceptor. bind (new InetSocketAddress (8080);} class TcpServerHandle extends IoHandlerAdapter {@ Override public void exceptionCaught (IoSession session, Throwable cause) throws Exception {cause. printStackTrace ();} // receives new data @ Override public void messageReceived (IoSession session, Object message) throws Exception {// receives client data, here, the receiving is no longer an IoBuffer type, but a String line = (String) message; System. out. println ("messageReceived:" + line) ;}@ Override public void sessionCreated (IoSession session) throws Exception {System. out. println ("sessionCreated") ;}@ Override public void sessionClosed (IoSession session) throws Exception {System. out. println ("sessionClosed ");}}

Netty:

The Netty design is similar to that of MINA. Some ChannelHandler needs to be added to ChannelPipeline to process raw data. LineBasedFrameDecoder is used to split the received data by line, and StringDecoder converts the data from bytecode to a string. Similarly, after the received data is processed, In the channelRead event function, the msg parameter is no longer a ByteBuf but a String.

Public class TcpServer {public static void main (String [] args) throws InterruptedException {EventLoopGroup bossGroup = new NioEventLoopGroup (); EventLoopGroup workerGroup = new NioEventLoopGroup (); try {ServerBootstrap B = new ServerBootstrap (); B. group (bossGroup, workerGroup ). channel (NioServerSocketChannel. class ). childHandler (new ChannelInitializer <SocketChannel> () {@ Override public void initChannel (SocketChannel ch) throws Exception {ChannelPipeline pipeline = ch. pipeline (); // LineBasedFrameDecoder splits the message pipeline by line. addLast (new LineBasedFrameDecoder (80); // encode it by UTF-8 into a string pipeline. addLast (new StringDecoder (CharsetUtil. UTF_8); pipeline. addLast (new TcpServerHandler () ;}}); ChannelFuture f = B. bind (8080 ). sync (); f. channel (). closeFuture (). sync ();} finally {workerGroup. shutdownGracefully (); bossGroup. shutdownGracefully () ;}} class TcpServerHandler extends ChannelInboundHandlerAdapter {// receives new data @ Override public void channelRead (ChannelHandlerContext ctx, Object msg) {// msg after StringDecoder, the type is no longer ByteBuf but String line = (String) msg; System. out. println ("channelRead:" + line) ;}@ Override public void channelActive (ChannelHandlerContext ctx) {System. out. println ("channelActive") ;}@ Override public void channelInactive (ChannelHandlerContext ctx) {System. out. println ("channelInactive") ;}@ Override public void exceptionCaught (ChannelHandlerContext ctx, Throwable cause) {cause. printStackTrace (); ctx. close ();}}

Twisted:

The Design of Twisted is not the same as that of the two above, so message splitting is not the same. The class TcpServerHandle for event processing no longer inherits Protocol, but inherits the LineOnlyReceiver subclass of Protocol. The event Method for receiving new data is not dataReceived, but lineReceived provided by LineOnlyReceiver. Looking at the Twisted source code, we can find that LineOnlyReceiver actually implements dataReceived internally and splits it by line. lineReceived is called when there is a new row of data.

#-*-Coding: UTF-8-*-from twisted. protocols. basic import LineOnlyReceiver from twisted. internet. protocol import Factory from twisted. internet import reactor class TcpServerHandle (LineOnlyReceiver): # create a new connection def connectionMade (self): print 'ononmade '# Disconnect def connectionLost (self, reason ): print 'connectionlost' # receives a new row of data def lineReceived (self, data): print 'linereceived: '+ data factory = Factory () factory. protocol = TcpServerHandle reactor. listenTCP (8080, factory) reactor. run ()

The following uses a Java client to test the three servers:

Public class TcpClient {public static void main (String [] args) throws IOException {Socket socket = null; OutputStream out = null; try {socket = new Socket ("localhost ", 8080); out = socket. getOutputStream (); // request the Server String lines = "Moonlight \ r \ n in front of the bed suspected to be frost on the ground \ r \ n raised his head and looked at the moon \ r \ n bowed his head to his hometown \ r \ n "; byte [] outputBytes = lines. getBytes ("UTF-8"); out. write (outputBytes); out. flush ();} finally {// close the connection out. close (); socket. close ();}}}

MINA server output result:

SessionCreated
MessageReceived: moonlight in front of bed
MessageReceived: suspected ground cream
MessageReceived
MessageReceived
SessionClosed

Netty server output result:

ChannelActive
ChannelRead: moonlight in front of bed
ChannelRead: suspected ground cream
ChannelRead
ChannelRead: shousi hometown
ChannelInactive

Twisted server output result:

ConnectionMade
LineReceived: moonlight in front of bed
LineReceived: suspected ground cream
LineReceived
LineReceived
ConnectionLost

Of course, the data sent during the test can also be simulated as not separated by rules. Next we will use a more abnormal client for testing:

Public class TcpClient {public static void main (String [] args) throws IOException, InterruptedException {Socket socket = null; OutputStream out = null; try {socket = new Socket ("localhost ", 8080); out = socket. getOutputStream (); String lines = "before bed"; byte [] outputBytes = lines. getBytes ("UTF-8"); out. write (outputBytes); out. flush (); Thread. sleep (1000); lines = ""; outputBytes = lines. getBytes ("UTF-8"); out. write (outputBytes); out. flush (); Thread. sleep (1000); lines = "light \ r \ n is suspected to be ground cream \ r \ n lifting"; outputBytes = lines. getBytes ("UTF-8"); out. write (outputBytes); out. flush (); Thread. sleep (1000); lines = "Hope moon \ r \ n bow thinking hometown \ r \ n"; outputBytes = lines. getBytes ("UTF-8"); out. write (outputBytes); out. flush ();} finally {// close the connection out. close (); socket. close ();}}}

Test the above three servers separately. The results are the same as the above output, so there is no problem.