Mina, Netty, twisted Study Together (v): Integration Protobuf

PROTOBUF is the abbreviation for Google's protocol buffers, which is used to convert between structured data and bytecode (serialization, deserialization), which is generally applied to network transmission and can support multiple programming languages.

Protobuf How to use here no longer introduced, this article is mainly introduced in Mina, Netty, twisted How to use PROTOBUF, do not know protobuf students can go to the exam I also have a blog post.

In a previous blog post. There is a method of message cutting that specifies the number of bytes of a body using a prefix header that is fixed to 4 bytes. Here the same to be used.

Only the content of the body is no longer a string, but a protobuf bytecode.

When dealing with business logic, you certainly don't want to serialize and deserialize the data. Instead, you want to manipulate an object directly, then you need to have a corresponding encoder and decoder. Writes the serialized and deserialized logic in the encoder and decoder. The implementation of encoders and decoders is described in the previous blog post.

The Netty package has a self-contained encoder and decoder for PROTOBUF. Then you don't have to do it for yourself. And Mina, twisted also need to implement their own PROTOBUF encoder and decoder.

This defines a PROTOBUF data structure that describes a student's information. Save As Studentmsg.proto file:

Message Student {    //ID    required Int32 id = 1;      Name    Required String name = 2;    Email    Optional String email = 3;    Friend    repeated string friends = 4;}

Generate Java and Python code separately using Studentmsg.proto. Add the code to the corresponding project.

The generated code is no longer posted.

The following describes how to use PROTOBUF to transmit student information in Netty, MINA, and twisted respectively.

Netty:

Netty comes with PROTOBUF encoder and decoder, each of which is protobufencoder and Protobufdecoder. It is important to note that Protobufencoder and Protobufdecoder are only responsible for PROTOBUF serialization and deserialization, Lengthfieldbasedframedecoder and Lengthfieldprepender are also required to handle message header prefixes and message cutting. The Lengthfieldbasedframedecoder is used to parse the message header prefix. Intercept Body,lengthfieldprepender, based on the number of body bytes specified in the header, is used to specify the number of body bytes by adding a header prefix to the message at the Wirte message.

public class TCPServer {public static void main (string[] args) throws Interruptedexception {Eventloopgroup Bos        Sgroup = new Nioeventloopgroup ();        Eventloopgroup Workergroup = new Nioeventloopgroup ();            try {serverbootstrap b = new Serverbootstrap (); B.group (Bossgroup, Workergroup). Channel (Nioserversocketchannel.class). Childhandler (New channelinitializer<socketchannel> () {@Override public void INITC Hannel (Socketchannel ch) throws Exception {Channelpipeline pipe  line = Ch.pipeline (); Responsible for using the body length specified by the 4-byte header to cut the message pipeline.addlast ("Framedecoder", New Lengthfieldbasedframedecoder (1048576, 0, 4, 0, 4)) ;//is responsible for the framedecoder processing of a complete message protobuf bytecode to student object Pipeline.addlast ("Protobufdecoder", New Protobufdecoder ( StudentMsg.Student.getDefaultInstance ()));//responsible for specifying the body length by adding a 4-byte header prefix to the written byte code Pipeline.addlasT ("Frameencoder", New Lengthfieldprepender (4));//responsible for turning student object into Protobuf bytecode pipeline.addlast ("Protobufencoder", new                            Protobufencoder ());                        Pipeline.addlast (New Tcpserverhandler ());            }                    });            Channelfuture f = b.bind (8080). sync ();        F.channel (). Closefuture (). sync ();            } finally {workergroup.shutdowngracefully ();        Bossgroup.shutdowngracefully (); }    }}

When handling an event, the received and sent parameters are directly the Student object:

public class Tcpserverhandler extends Channelinboundhandleradapter {@Override public void Channelread (ChannelHandle        Rcontext ctx, Object msg) {//Read Student object passed by client studentmsg.student Student = (studentmsg.student) msg;        System.out.println ("ID:" + Student.getid ());        System.out.println ("Name:" + student.getname ());        System.out.println ("Email:" + student.getemail ());        System.out.println ("Friends:");        List<string> friends = Student.getfriendslist ();        for (String friend:friends) {System.out.println (friend);        }//Create a new Student object to upload to client StudentMsg.Student.Builder Builder = StudentMsg.Student.newBuilder ();        Builder.setid (9);        Builder.setname ("server");        Builder.setemail ("[email protected]");        Builder.addfriends ("X");        Builder.addfriends ("Y");        Studentmsg.student Student2 = Builder.build ();    Ctx.writeandflush (Student2); } @Override public void EXceptioncaught (Channelhandlercontext ctx, throwable cause) {cause.printstacktrace ();    Ctx.close (); }}

MINA:

There are no encoders and decoders for Protobuf in the Mina. But you can implement a function and Netty the same encoder and decoder.

Encoder:

public class Minaprotobufencoder extends Protocolencoderadapter {    @Override public    void Encode (iosession Session, Object message,            protocolencoderoutput out) throws Exception {    studentmsg.student Student = ( studentmsg.student) message;        byte[] bytes = Student.tobytearray (); Student object to protobuf byte code        int length = bytes.length;                Iobuffer buffer = iobuffer.allocate (length + 4);        Buffer.putint (length); Write header        buffer.put (bytes);//write Body        buffer.flip ();        Out.write (buffer);}    }

Decoder:

public class Minaprotobufdecoder extends Cumulativeprotocoldecoder {@Overrideprotected Boolean dodecode (iosession Session, Iobuffer In,protocoldecoderoutput out) throws Exception {//assuming that the header portion (4 bytes) is not received. return directly to Falseif (In.remaining () < 4) {return false;} else {//Mark start position, if a message is not transferred, return to this position in.mark ();//Read header section, Gets the body length int bodylength = In.getint ();//Assuming the body is not received intact, return directly to Falseif (In.remaining () < Bodylength) {in.reset (); Iobuffer position return to the original marked place return false; else {byte[] bodybytes = new Byte[bodylength];in.get (bodybytes);//Read body part studentmsg.student Student = StudentMsg.Student.parseFrom (bodybytes); Convert the Protobuf byte code in body into student object Out.write (student); Parse out a message return true;}}}

Minaserver Add encoder and decoder to PROTOBUF:

public class TCPServer {public    static void Main (string[] args) throws IOException {        Ioacceptor acceptor = new Nio Socketacceptor ();        Specifies the PROTOBUF encoder and Decoder        acceptor.getfilterchain (). AddLast ("Codec",                new Protocolcodecfilter (new Minaprotobufencoder (), New Minaprotobufdecoder ());        Acceptor.sethandler (New Tcpserverhandle ());        Acceptor.bind (new inetsocketaddress (8080));}    }

Such When dealing with business logic, it is the same as Netty:

public class Tcpserverhandle extends Iohandleradapter {@Override public void Exceptioncaught (iosession session, THR    Owable cause) throws Exception {cause.printstacktrace (); } @Override public void messagereceived (iosession session, Object message) throws Exception {//Read        Student object Studentmsg.student Student = (studentmsg.student) message sent by client;        System.out.println ("ID:" + Student.getid ());        System.out.println ("Name:" + student.getname ());        System.out.println ("Email:" + student.getemail ());        System.out.println ("Friends:");        List<string> friends = Student.getfriendslist ();        for (String friend:friends) {System.out.println (friend);        }//Create a new Student object to upload to client StudentMsg.Student.Builder Builder = StudentMsg.Student.newBuilder ();        Builder.setid (9);        Builder.setname ("server");        Builder.setemail ("[email protected]"); Builder.addfriends ("X");        Builder.addfriends ("Y");        Studentmsg.student Student2 = Builder.build ();    Session.write (Student2); }}

Twisted:

In the twisted. First, define a Protobufprotocol class that inherits the Protocol class and acts as an encoder and decoder. The Tcpserverhandle class that processes the business logic inherits the Protobufprotocol class. Invokes or overrides the method provided by the Protobufprotocol.

#-*-coding:utf-8–*-from struct import pack, Unpackfrom twisted.internet.protocol import Factoryfrom TWISTED.INTERNET.P Rotocol Import protocolfrom twisted.internet import reactorimport studentmsg_pb2# protobuf encoding, decoder class Protobufprotocol (Protocol): # Used to temporarily store received data _buffer = B "" Def datareceived (Self, data): # The last unhandled data plus the data received this time Self._b Uffer = self._buffer + data # keeps looping until the new message does not receive full while True: # Assume that the header receives the full if Len (self.                 _buffer) >= 4: # header section, press the large byte order to int, get body length lengths, = Unpack (">i", Self._buffer[0:4])                     # Suppose the body receives full if Len (Self._buffer) >= 4 + Length: # body part, Protobuf byte code                    Packet = self._buffer[4:4 + length] # protobuf byte code to student object Student = Studentmsg_pb2. Student () Student. Parsefromstring (packet) # call ProtoBufreceived incoming Student Object self.protobufreceived (student) # Remove _buffe            The message part that has been processed in R self._buffer = self._buffer[4 + length:] else:break;    Else:break; def protobufreceived (self, Student): Raise Notimplementederror def sendprotobuf (self, Student): # Student The object is converted to PROTOBUF byte code data = student. Serializetostring () # Add header prefix to specify PROTOBUF bytecode length Self.transport.write (Pack (">i", Len (data) + data) # logic code C         Lass Tcpserverhandle (Protobufprotocol): # implements the Protobufreceived def protobufreceived (self, student) provided by Protobufprotocol: # will receive student output print ' ID: ' + str (student.id) print ' Name: ' + student.name print ' Email: ' + Student.email print ' Friends: ' for friend in Student.friends:print friend # creates a student concurrency Send to client Student2 = Studentmsg_pb2. Student () student2.iD = 9 Student2.name = ' server '. Decode (' UTF-8 ') # Chinese needs to be converted to UTF-8 string student2.email = ' [email protected] ' Student2.friends.append (' X ') student2.friends.append (' Y ') self.sendprotobuf (student2) factory = Factory ( ) Factory.protocol = tcpserverhandlereactor.listentcp (8080, Factory) Reactor.run ()

The following is a client test program written in Java:

public class TcpClient {public static void main (string[] args) throws IOException {socket socket = NULL;        DataOutputStream out = null;                DataInputStream in = null;            try {socket = new socket ("localhost", 8080);            out = new DataOutputStream (Socket.getoutputstream ());                        in = new DataInputStream (Socket.getinputstream ());            Create a Student passed to server StudentMsg.Student.Builder Builder = StudentMsg.Student.newBuilder ();            Builder.setid (1);            Builder.setname ("Client");            Builder.setemail ("[email protected]");            Builder.addfriends ("A");            Builder.addfriends ("B");            Studentmsg.student Student = Builder.build (); byte[] outputbytes = Student.tobytearray (); Student to byte code out.writeint (Outputbytes.length); Write Header Out.write (outputbytes);                        Write body Out.flush (); Get SerVer passed over the student int bodylength = In.readint ();            Read header byte[] bodybytes = new Byte[bodylength];  In.readfully (bodybytes); Read body studentmsg.student Student2 = StudentMsg.Student.parseFrom (bodybytes);            The body byte code is parsed into student System.out.println ("Header:" + bodylength);            System.out.println ("Body:");            System.out.println ("ID:" + Student2.getid ());            System.out.println ("Name:" + student2.getname ());            System.out.println ("Email:" + student2.getemail ());            System.out.println ("Friends:");            List<string> friends = Student2.getfriendslist ();            for (String friend:friends) {System.out.println (friend);            }} finally {//close connection in.close ();            Out.close ();        Socket.close (); }    }}

Test each of the above three tcpserver with the client:

Server output:

Id:1
Name:client
Email:[email protected]
Friends:
A
B

Client output:

Header:32
Body:
Id:9
Name:server
Email:[email protected]
Friends:
X
Y

Fork Brother reproduced please indicate the source: http://blog.csdn.net/xiao__gui/article/details/38864961

MINA, Netty, twisted together study series

MINA, Netty, twisted learn Together (i): Achieve a simple tcpserver

MINA, Netty, twisted Together (ii): TCP message boundary problems and cut messages by row

MINA, Netty, Twisted learn Together (iii): TCP message fixed-size prefix (Header)

MINA, Netty, Twisted learn Together (iv): Customize your own protocol

MINA, Netty, twisted Study Together (v): Integration Protobuf

MINA, Netty, twisted study Together (VI): Session

MINA, Netty, twisted Together (vii): Announcement/Subscription (Publish/subscribe)

MINA, Netty, twisted Together (eight): Httpserver

MINA, Netty, Twisted learn Together (ix): Asynchronous IO and callback functions

MINA, Netty, Twisted learn together (10): Threading model

MINA, Netty, Twisted learn together (11): SSL/TLS

MINA, Netty, Twisted learn together (12): HTTPS

Source

https://github.com/wucao/mina-netty-twisted

Mina, Netty, twisted Study Together (v): Integration Protobuf