Understanding Netty encoding and decoding of protocol buffers

One, Netty+protocol buffers brief description

Netty is one of the most popular NIO frameworks in the industry
Advantages:
1) Easy to use API, low development threshold;
2) Powerful, preset a variety of codec functions, support a variety of mainstream protocols;
3) Strong customization ability, the communication framework can be flexibly expanded through Channelhandler;
4) High performance, through the comparison with other industry mainstream NIO framework, Netty of the comprehensive energy optimization;
5) Mature, stable, netty fixed all of the JDK NiO bugs that have been found, and business developers don't need to worry about nio bugs anymore;
6) The community is active, the version iteration cycle is short, the bugs found can be repaired in time, and more new features will be added;
7) experienced a large-scale commercial application test, quality has been verified. In the Internet, big data, online games, enterprise applications, telecommunications software and many other industries have been successful commercial, it proves that it can fully meet the commercial applications of different industries.

Protocol buffers is a way for Google to encode structured data in an efficient and extensible format.
It can be used for data storage, communication protocol, and so on, it is not dependent on language and platform, and is highly extensible.
At this stage, the official support of C + +, JAVA, Python and other three programming languages, but can find a large number of almost all languages of the third-party expansion package.

Advantages
1) binary message, good performance/high efficiency (space and time efficiency are very good)
2) proto file generates target code, easy to use
3) Serialization of the data classes in the direct corresponding program, do not need to parse after the mapping (Xml,json is this way)
4) Support for forward compatibility (new plus fields with default values) and backward compatibility (ignoring new fields) simplifies upgrade
5) Support multiple languages

Second, know Varint

Proto message format such as: Length + protobuf data (message header + message)

The message header describes the length of the message data body. To reduce the amount of transmission, the message header is in varint format.

What is Varint?

Varint is a compact way to represent numbers. It uses one or more bytes to represent a number, and the smaller the number, the smaller the number of bytes. This reduces the number of bytes used to represent the number.
The highest bit of each byte in the varint has a special meaning, if the bit is 1, the subsequent byte is also part of the number, and if the bit is 0, the end. The other 7 bits are used to represent numbers. Therefore, a number less than 128 can be represented by a byte. A number greater than 128 will use two bytes.

For example, the representation of an integer 1 requires only one byte:

0000 0001

For example, a 300 representation requires two bytes:

1010 1100 0000 0010

With Varint, for a small number of int32 types, it can be represented by 1 bytes. Of course everything has good and bad side, using varint notation, large numbers need 5 byte to represent. From a statistical point of view, generally not all of the numbers in the message are large numbers, so in most cases, with varint, you can use a smaller number of bytes to represent the digital information.

Demonstrates how Google Protocol Buffer resolves two bytes. Note that the position of the two byte is exchanged once before the final calculation, because the Google Protocol Buffer byte order takes the form of a Little-endian.

Third, the encoding and decoding of communication

Netty provides support for protocol buffers by default, so it's easy to integrate. The key to an integrated relationship is the understanding of encoding and decoding.

2.11 Server-side the basic code for a service is as follows:

eventloopgroup boosgroup = new Nioeventloopgroup (); Eventloopgroup WorkGroup = new Nioeventloopgroup (); try {serverbootstrap bootstrap = new Serverbootstrap (); Bootstrap.group (Boosgroup, WorkGroup). Channel (nioserversocketchannel.class). Option (Channeloption.so_backlog, 100 ). Childhandler (New channelinitializer<channel> () {@Overrideprotected void Initchannel (Channel ch) throws Exception {ch.pipeline (). AddLast (New Protobufvarint32framedecoder ());//decode (process half package) Ch.pipeline (). AddLast (New Protobufdecoder (MsgProto.Packet.getDefaultInstance ())); Ch.pipeline (). AddLast (New Protobufvarint32lengthfieldprepender ());//Add Length Ch.pipeline (). AddLast (New Protobufencoder ());//Code Ch.pipeline (). AddLast (New Serverchannelhandleradapter ());//Business Process Handler});//bind port channelfuture future = Bootstrap.bind. Sync ();//wait to close Future.channel (). Closefuture (). sync ();} catch (Exception e) {Log.error ("{}", e);} finally {boosgroup.shutdowngracefully (); workgroup.shutdowngracefully ();} 

　　

2.21 focus on encoding and decoding source class,Protobufvarint32framedecoder,protobufvarint32lengthfieldprepender,protobufencoder

Encoding class:

Protobufencoder: This just writes the Proto object directly to the out

@Sharablepublic class Protobufencoder extends messagetomessageencoder<messageliteorbuilder> {    @Override    protected void Encode (            channelhandlercontext ctx, Messageliteorbuilder msg, list<object> out) throws Exception {        if (msg instanceof messagelite) {            Out.add (Wrappedbuffer (((messagelite) msg). Tobytearray ()));            return;        }        if (msg instanceof messagelite.builder) {            Out.add (Wrappedbuffer (((Messagelite.builder) msg). Build (). Tobytearray ()));}}}    

The key is the Protobufvarint32lengthfieldprepender class:

* before DECODE (bytes) after DECODE (302 bytes)
* +---------------+ +--------+---------------+
* | Protobuf Data |-------------->| Length | Protobuf Data |
* | (bytes) | | 0XAC02 | (bytes) |
* +---------------+ +--------+---------------+

Data format = data Length (header) + Real data @sharablepublic class Protobufvarint32lengthfieldprepender extends messagetobyteencoder< bytebuf> {    @Override    protected void encode (            channelhandlercontext ctx, Bytebuf msg, bytebuf out) throws Exception {        int bodylen = Msg.readablebytes ();        int headerlen = codedoutputstream.computerawvarint32size (Bodylen);        Out.ensurewritable (Headerlen + bodylen);//Make sure the writable length (head length + data length)        //Data length is written to the head        codedoutputstream headerout =                Codedoutputstream.newinstance (New Bytebufoutputstream (out), Headerlen);        Headerout.writerawvarint32 (Bodylen);        Headerout.flush ();        Write Data        out.writebytes (MSG, Msg.readerindex (), Bodylen);}    }

　　

the corresponding decoding class Protobufvarint32framedecoder:

Things to do like annotations:

* Before DECODE (302 bytes) after DECODE (bytes)
* +--------+---------------+ +---------------+
* | Length | Protobuf Data |----->| Protobuf Data |
* | 0xac02 | (bytes) | | (bytes) |
* +--------+---------------+ +---------------+

Half-pack sticky bag handling public class Protobufvarint32framedecoder extends Bytetomessagedecoder {//TODO Maxframelength + safe Skip + F Ail-fast option//(Just like lengthfieldbasedframedecoder) @Override protected void decode (Channelhandlerco ntext CTX, bytebuf in, list<object> out) throws Exception {In.markreaderindex ();//Mark Read position final byte[                 ] buf = new Byte[5];//varint32 maximum 5 bytes for (int i = 0; i < buf.length; i + +) {if (!in.isreadable ()) {                In.resetreaderindex ();            Return            } Buf[i] = In.readbyte (); if (Buf[i] >= 0) {int length = codedinputstream.newinstance (buf, 0, i + 1). ReadRawVarint32 ();//varint  Format to actual length int if (length < 0) {throw new corruptedframeexception ("Negative length:"                + length);             } if (In.readablebytes () < length) {//Length not enough, rollback tag in.resetreaderindex ();       Return                } else {Out.add (in.readbytes (length));//correct read returns return;        }}}//couldn ' t find the byte whose MSB is off.    throw new Corruptedframeexception ("length wider than 32-bit"); }}

　　

Four, summary

Netty Default has helped us achieve the protocol buffers codec, so it is very convenient to use.

However, if you have special needs, such as encryption, you need to understand the encoding and decoding rules and reference its default implementation.

Understanding Netty encoding and decoding of protocol buffers