Mina, Netty, and Twisted: Integrating protobuf and nettyprotobuf

Mina, Netty, and Twisted: Integrating protobuf and nettyprotobuf

Protobuf is short for Google's Protocol Buffers. It is used for the conversion (serialization and deserialization) between structured data and bytecode. It is generally used for network transmission and supports multiple programming languages.

This article mainly describes how to use protobuf in MINA, Netty, and Twisted. If you are not familiar with protobuf, you can refer to another blog.

In the previous blog, we introduced a message segmentation method that uses a fixed 4-byte prefix Header to specify the number of bytes of the Body, it should also be used here. The content of the Body is not a string, but a protobuf bytecode.

When processing the business logic, you certainly do not want to serialize or deserialize the data, but want to directly operate an object. Therefore, you must have the corresponding encoder and decoder, write the serialization and deserialization logic in the encoder and decoder. The implementation of encoder and decoder is described in the previous blog.

The Netty package already comes with the encoder and decoder for protobuf, so you don't have to implement it yourself. MINA and Twisted also need to implement protobuf encoder and decoder by themselves.

A protobuf data structure is defined here to describe the information of a student and save it as the StudentMsg. proto file:

Message Student {// ID required int32 id = 1; // name required string name = 2; // email optional string email = 3; // friend repeated string friends = 4 ;}

Use StudentMsg. proto to generate Java and Python code respectively and add the code to the corresponding project. The generated code is no longer pasted.

The following describes how Netty, MINA, and Twisted use protobuf to transmit Student information.

Netty:

Netty comes with protobuf encoder and decoder, which are ProtobufEncoder and ProtobufDecoder. It should be noted that ProtobufEncoder and ProtobufDecoder are only responsible for protobuf serialization and deserialization, while LengthFieldBasedFrameDecoder and LengthFieldPrepender are also required to process the message Header prefix and message segmentation. LengthFieldBasedFrameDecoder is used to parse the message Header prefix. The Body is intercepted based on the number of Body bytes specified in the Header. LengthFieldPrepender is used to add a Header prefix before the message in wirte message to specify the number of Body bytes.

Public class TcpServer {public static void main (String [] args) throws InterruptedException {EventLoopGroup bossGroup = new NioEventLoopGroup (); EventLoopGroup workerGroup = new NioEventLoopGroup (); try {ServerBootstrap B = new ServerBootstrap (); B. group (bossGroup, workerGroup ). channel (NioServerSocketChannel. class ). childHandler (new ChannelInitializer <SocketChannel> () {@ Override public void initChannel (SocketChannel ch) throws Exception {ChannelPipeline pipeline = ch. pipeline (); // cut the message by using the Body length specified by the 4-byte Header. addLast ("frameDecoder", new LengthFieldBasedFrameDecoder (1048576, 0, 4, 0, 4 )); // converts the protobuf bytecode of a complete message processed by frameDecoder to the pipeline object of Student. addLast ("protobufDecoder", new ProtobufDecoder (StudentMsg. student. getDefaultInstance (); // specifies the Body length pipeline by adding the 4-byte Header prefix to the written bytecode. addLast ("frameEncoder", new LengthFieldPrepender (4); // converts a Student object to a protobuf bytecode pipeline. addLast ("protobufEncoder", new ProtobufEncoder (); pipeline. addLast (new TcpServerHandler () ;}}); ChannelFuture f = B. bind (8080 ). sync (); f. channel (). closeFuture (). sync ();} finally {workerGroup. shutdownGracefully (); bossGroup. shutdownGracefully ();}}}

When processing an event, the received and sent parameters are directly Student objects:

Public class TcpServerHandler extends ChannelInboundHandlerAdapter {@ Override public void channelRead (ChannelHandlerContext ctx, Object msg) {// read the StudentMsg Object passed by the client. student student = (StudentMsg. student) msg; System. out. println ("ID:" + student. getId (); System. out. println ("Name:" + student. getName (); System. out. println ("Email:" + student. getEmail (); System. out. println ("Friends:"); List <String> friends = student. getFriendsList (); for (String friend: friends) {System. out. println (friend);} // create a new Student object and upload it to the client StudentMsg. student. builder builder = StudentMsg. student. newBuilder (); builder. setId (9); builder. setName ("server"); builder. setEmail ("123@abc.com"); builder. addFriends ("X"); builder. addFriends ("Y"); StudentMsg. student student2 = builder. build (); ctx. writeAndFlush (student2);} @ Override public void exceptionCaught (ChannelHandlerContext ctx, Throwable cause) {cause. printStackTrace (); ctx. close ();}}

MINA:

There is no encoder or decoder for protobuf in MINA, but you can implement an encoder and decoder with the same functions as Netty.

Encoder:

Public class MinaProtobufEncoder extends ProtocolEncoderAdapter {@ Override public void encode (IoSession session, Object message, ProtocolEncoderOutput out) throws Exception {StudentMsg. student student = (StudentMsg. student) message; byte [] bytes = student. toByteArray (); // convert the Student object to the protobuf bytecode int length = bytes. length; IoBuffer buffer = IoBuffer. allocate (length + 4); buffer. putInt (length); // write header buffer. put (bytes); // write body buffer. flip (); out. write (buffer );}}

Decoder:

Public class MinaProtobufDecoder extends CumulativeProtocolDecoder {@ Override protected boolean doDecode (IoSession session, IoBuffer in, ProtocolDecoderOutput out) throws Exception {// If the Header is not received (4 bytes ), returns false if (in. remaining () <4) {return false;} else {// mark the start position. If a message is not transmitted completely, it is returned to this position in. mark (); // read the header and obtain the body length int bodyLength = in. getInt (); // returns false if (in. remaining () <bodyLength) {in. reset (); // return the position of IoBuffer to the place where it was originally marked; return false;} else {byte [] bodyBytes = new byte [bodyLength]; in. get (bodyBytes); // read the body part StudentMsg. student student = StudentMsg. student. parseFrom (bodyBytes); // convert the protobuf bytecode in the body to the Student object out. write (student); // parse a message return true ;}}}}

The MINA server is added to the protobuf encoder and decoder:

Public class TcpServer {public static void main (String [] args) throws IOException {IoAcceptor acceptor = new NioSocketAcceptor (); // specify the encoder and decoder acceptor of protobuf. getFilterChain (). addLast ("codec", new ProtocolCodecFilter (new MinaProtobufEncoder (), new MinaProtobufDecoder (); acceptor. setHandler (new TcpServerHandle (); acceptor. bind (new InetSocketAddress (8080 ));}}

In this way, when processing the business logic, it is the same as Netty:

Public class TcpServerHandle extends IoHandlerAdapter {@ Override public void exceptionCaught (IoSession session, Throwable cause) throws Exception {cause. printStackTrace () ;}@ Override public void messageReceived (IoSession session, Object message) throws Exception {// read the Student Object StudentMsg passed by the client. student student = (StudentMsg. student) message; System. out. println ("ID:" + student. getId (); System. out. println ("Name:" + student. getName (); System. out. println ("Email:" + student. getEmail (); System. out. println ("Friends:"); List <String> friends = student. getFriendsList (); for (String friend: friends) {System. out. println (friend);} // create a new Student object and upload it to the client StudentMsg. student. builder builder = StudentMsg. student. newBuilder (); builder. setId (9); builder. setName ("server"); builder. setEmail ("123@abc.com"); builder. addFriends ("X"); builder. addFriends ("Y"); StudentMsg. student student2 = builder. build (); session. write (student2 );}}

Twisted:

In Twisted, A ProtobufProtocol class is defined to inherit the Protocol Class and act as the encoder and decoder. The TcpServerHandle class that processes the business logic inherits the ProtobufProtocol class and calls or overrides the methods provided by ProtobufProtocol.

#-*-Coding: UTF-8-*-from struct import pack, unpack from twisted. internet. protocol import Factory from twisted. internet. protocol import Protocol from twisted. internet import reactor import maid # protobuf encoding, decoder class ProtobufProtocol (Protocol): # used to temporarily store received data _ buffer = B "" def dataReceived (self, data ): # The Last unprocessed data plus the received data self. _ buffer = self. _ buffer + data # keep repeating until the new message does not receive the complete while True: # if the header receives the complete if len (self. _ buffer)> = 4: # header part, convert it to int In the byte order, get the body length, = unpack ("> I", self. _ buffer [0: 4]) # if the body receives the complete if len (self. _ buffer)> = 4 + length: # body part, protobuf bytecode packet = self. _ buffer [4: 4 + length] # convert protobuf bytecode to Student object student = StudentMsg_pb2.Student () student. parseFromString (packet) # Call protobufReceived to input the Student object self. protobufReceived (student) # Remove the message that has been processed in _ buffer. _ buffer = self. _ buffer [4 + length:] else: break; def protobufReceived (self, student): raise NotImplementedError def sendProtobuf (self, student ): # convert Student object to protobuf bytecode data = student. serializeToString () # Add the Header prefix to specify the protobuf bytecode length self. transport. write (pack ("> I", len (data) + data) # logic code class TcpServerHandle (ProtobufProtocol): # implement protobufReceived def protobufReceived (self, student) provided by ProtobufProtocol ): # print 'id: '+ str (Student. id) print 'name: '+ student. name print 'email: '+ student. email print 'friends: 'for friend in student. friends: print friend # create a Student and send it to the client student2 = StudentMsg_pb2.Student () student2.id = 9 student2.name = 'server '. decode ('utf-8') # Chinese needs to be converted into a UTF-8 string student2.email = '2017 @ abc.com 'student2.friends. append ('x') student2.friends. append ('y') self. sendProtobuf (student2) factory = Factory () factory. protocol = TcpServerHandle reactor. listenTCP (8080, factory) reactor. run ()

The following is a client test program written in Java:

Public class TcpClient {public static void main (String [] args) throws IOException {Socket socket = null; DataOutputStream out = null; DataInputStream in = null; try {socket = new Socket ("localhost", 8080); out = new DataOutputStream (socket. getOutputStream (); in = new DataInputStream (socket. getInputStream (); // create a Student and send it to the server StudentMsg. student. builder builder = StudentMsg. student. newBuilder (); builder. setId (1); builder. setName ("client"); builder. setEmail ("xxg@163.com"); builder. addFriends ("A"); builder. addFriends ("B"); StudentMsg. student student = builder. build (); byte [] outputBytes = student. toByteArray (); // Student is converted into a bytecode out. writeInt (outputBytes. length); // write header out. write (outputBytes); // write body out. flush (); // get the Student int bodyLength = in. readInt (); // read header byte [] bodyBytes = new byte [bodyLength]; in. readFully (bodyBytes); // read body StudentMsg. student student2 = StudentMsg. student. parseFrom (bodyBytes); // The body bytecode is parsed to Student System. out. println ("Header:" + bodyLength); System. out. println ("Body:"); System. out. println ("ID:" + student2.getId (); System. out. println ("Name:" + student2.getName (); System. out. println ("Email:" + student2.getEmail (); System. out. println ("Friends:"); List <String> friends = student2.getFriendsList (); for (String friend: friends) {System. out. println (friend) ;}} finally {// close the connection in. close (); out. close (); socket. close ();}}}

Use the client to test the preceding three TCP servers:

Server output:

ID: 1
Name: Client
Email: xxg@163.com
Friends:
A
B

Client output:

Header: 32
Body:
ID: 9
Name: Server
Email: 123@abc.com
Friends:
X
Y

Who has spring integrated hibernate project, cannot use struts framework, if there is a servlet to provide data through mina or netty Communication

Sent to your mailbox