Custom PROTOBUF Codec

PROTOBUF3 + Netty4: Transmitting multiple types of PROTOBUF data on the socket

PROTOBUF serialized byte stream data is not self-describing, and when we send data to the client via the socket, the client must know what type of data is being sent in order to deserialize it correctly. This severely affects the implementation of the C/S function, which is in fact transmitted only one type of data. This article explains the solution I used, although I think there should be an official implementation of the more reasonable, that is, the native support protobuf of the self-describing.

(In the Financial field, there is a protocol called fast that has the same fundamentals and Protobuf, and has a higher compression rate, and the serialized byte stream is self-describing, The data that can be automatically deserialized into the corresponding template (the template corresponds to the. proto file), but the time efficiency is worse than the protobuf, we can also pay attention to. ）

Solution One

First, introduce another implementation, a workaround described in the official wiki of protobuf, by defining a type for self-describing:

Message Selfdescribingmessage {
  //Set of. Proto files which define the type.
  Required Filedescriptorset proto_files = 1;

  Name of the message type.  must be defined by one of the "files in
  //proto_files."
  Required String type_name = 2;

  The message data.
  Required bytes Message_data = 3;
}

(Reference: Https://developers.google.com/protocol-buffers/docs/techniques#self-description)

Place the byte array of the type actually to be transferred in the Message_data field, describing its proto file and type with the Proto_files and Type_name fields. Thus, the Selfdescribingmessage type is transmitted over the channel, but the payload on it can be any type of data.

I have not tried this way. The reason I'm not willing to use this way is obviously that it requires 2 serialization and 2 deserialization, and the byte array is created 2 times. This is not good enough if it corresponds to a system that is sensitive to delay and performance.

Solution Two

The main plan to be introduced today. In front of the PROTOBUF serialization, add a custom header that contains the length of the serialization and its type. Deserialization is based on the header at the time of decompression.

If you want to transfer 2 types of data on the socket, stock quotes information and options Market information:

A. Proto definition of a stock:

Syntax = "Proto3";

Package test.model.protobuf;

Option Java_package = "Test.model.protobuf";

Message Stocktick {
    string stockid = 1;
    int price = 2;
}

The. Proto definition of an option:

Syntax = "Proto3";

Package test.model.protobuf;

Option Java_package = "Test.model.protobuf";

Message Optiontick {
    string optionid = 1;
    string Securityid = 2;
    int price = 3;
}

NETTY4 official has actually implemented the PROTOBUF codec plug-in, but only for the transmission of a single type of PROTOBUF serialization. I am here to intercept a Netty code, familiar with Netty students can immediately understand its role:

        @Override
        protected void Initchannel (Socketchannel ch) throws Exception {Channelpipeline
            pipeline = Ch.pipeline ();
            Pipeline.addlast (New Protobufvarint32framedecoder ());
            Pipeline.addlast (New Protobufdecoder (StockTickOuterClass.StockTick.getDefaultInstance ()));
            Pipeline.addlast (New Protobufvarint32lengthfieldprepender ());
            Pipeline.addlast (New Protobufencoder ());
            Pipeline.addlast (New Customprotoserverhandler ());
        }

Look at the above code highlighting section, Netty4 the official codec must specify a single protobuf type. Specific functions of each class:

Protobufencoder: For serialization of probuf types.
Protobufvarint32lengthfieldprepender: Used to precede the serialized byte array with a simple header, containing only the serialized byte length.
Protobufvarint32framedecoder: Used to solve half-pack and sticky-packet problems before decode (using the length of the array contained in the header to identify half-packet sticky packets)
Protobufdecoder: Deserializes the specified probuf byte array as the PROTOBUF type.

We can refer to the above official codec code, will implement our customized PROTOBUF codec plug-in, but to support a variety of different types of PROTOBUF data on a socket transmission:

Encoder Customprotobufencoder:

Import Com.google.protobuf.MessageLite;
Import Io.netty.buffer.ByteBuf;
Import io.netty.channel.ChannelHandler.Sharable;
Import Io.netty.channel.ChannelHandlerContext;

Import Io.netty.handler.codec.MessageToByteEncoder; /** * Reference Protobufvarint32lengthfieldprepender and Protobufencoder * * @Sharable public class Customprotobufencoder extends M
    
    essagetobyteencoder<messagelite> {Hangqingencoder hangqingencoder;
    Public Customprotobufencoder (Hangqingencoder hangqingencoder) {this.hangqingencoder = Hangqingencoder; @Override protected void Encode (Channelhandlercontext ctx, Messagelite msg, bytebuf out) throw
        S Exception {byte[] body = Msg.tobytearray ();
        
        Byte[] Header = Encodeheader (msg, (short) body.length);
        Out.writebytes (header);
        
        Out.writebytes (body);
    Return Private byte[] Encodeheader (messagelite msg, short bodylength) {byte Messagetype = 0x0f;
        if (msg instanceof stocktickouterclass.stocktick) {messagetype = 0x00;
        else if (msg instanceof optiontickouterclass.optiontick) {messagetype = 0x01;
        } byte[] Header = new Byte[4];
        Header[0] = (byte) (Bodylength & 0xff);
        HEADER[1] = (byte) ((Bodylength >> 8) & 0xff); HEADER[2] = 0;

        Reserved field header[3] = MessageType;

    return header; }
}

Customprotobufencoder serializes the incoming PROTOBUF type and creates a 4-byte header for it, formatted as follows

Body Length (Low)

Body length (high)

Reserved bytes

Type

The implementation of the Encodeheader method is based on what type of protobuf you want to transfer to modify the code, or a little design to avoid using too many if...else.

Decoder Customprotobufdecoder:

Import Io.netty.buffer.ByteBuf;
Import Io.netty.channel.ChannelHandlerContext;
Import Io.netty.handler.codec.ByteToMessageDecoder;
Import java.util.List;

Import Com.google.protobuf.MessageLite; /** * Reference Protobufvarint32framedecoder and Protobufdecoder * * public class Customprotobufdecoder extends Bytetomessagedecod Er {@Override protected void decode (Channelhandlercontext ctx, bytebuf in, list<object> out) throws
            Ion {while (In.readablebytes () > 4) {//If the readable length is less than the header length, exit.

            In.markreaderindex ();
            Gets the body length byte low = In.readbyte () in the header;
            byte high = In.readbyte ();
            Short s0 = (short) (Low & 0xff);
            Short S1 = (short) (High & 0xff);
            S1 <<= 8;

            Short length = (short) (S0 | s1);
            Gets the Protobuf type In.readbyte () in the Baotou;

            byte DataType = In.readbyte ();
            If the readable length is less than the body length, resume the read pointer and exit. if (In.readablebytes () < length) {In.resetreaderindex ();
            Return }//Read body bytebuf bodybytebuf =