Netty decoding and Packaging research on PROTOBUF Protocol (2)

Netty Default support PROTOBUF encapsulation and decoding, if both sides of the communication use Netty there is no obstacle, but if the client is another language (C #) You need to write their own copy and Netty consistent way (decoding + encapsulation), In advance it is important to understand how netty is encapsulated and decoded. Here mainly by reading the source code main class Protobufvarint32framedecoder (decoding) +protobufvarint32lengthfieldprepender (encapsulation) to resolve its principle and implementation.

Article source Http://www.cnblogs.com/tankaixiong

One, support the default implementation of the PROTOBUF protocol

//Configuring the service-side NIO thread GroupEventloopgroup Bossgroup =NewNioeventloopgroup (); Eventloopgroup Workergroup=NewNioeventloopgroup (); Try{Serverbootstrap b=NewServerbootstrap (); B.group (Bossgroup, Workergroup). Channel (Nioserversocketchannel.class). Option (Channeloption.so_backlog,1024). Handler (NewLogginghandler (Loglevel.info)). Childhandler (NewChannelinitializer<socketchannel>() {@Overrideprotected voidInitchannel (Socketchannel ch)throwsException {ch.pipeline (). AddLast (NewProtobufvarint32framedecoder ()). AddLast (NewProtobufdecoder (SubscribeReqProto.SubscribeReq.getDefaultInstance ())) . AddLast (NewProtobufvarint32lengthfieldprepender ()). AddLast (NewProtobufencoder ()). AddLast (NewSubreqserverhandler ());              }                                        }); //bind port, synchronization waits for successChannelfuture f =B.bind (port). sync (); //wait for the server to listen port shutdownF.channel (). Closefuture (). sync (); }finally{              //freeing resources when exitingbossgroup.shutdowngracefully ();          Workergroup.shutdowngracefully (); }

The above is the default implementation provided. The key is the Protobufvarint32framedecoder,protobufvarint32lengthfieldprepender class.

Second, Protobufvarint32lengthfieldprepender coding class

An encoder that prepends the the the Google Protocol buffers varints integer Length field.

* BEFORE DECODE (300 bytes)       AFTER DECODE (302 bytes)* +---------------+               +--------+---------------+* | Protobuf Data |-------------->| Length | Protobuf Data |* | (300 bytes) | | 0xAC02 | (300 bytes) |* +---------------+ +--------+---------------+

From the description of the class, proto message format, such as: Length + protobuf data (message header + message) mode, it is particularly important to note that the head length is used varints method is not an int, the message header describes the length of the message data body. To reduce the amount of transmission, the message header is in varint format.

What is Varint?

Article source Http://www.cnblogs.com/tankaixiongVarint is a compact way to represent numbers. It uses one or more bytes to represent a number, and the smaller the number, the smaller the number of bytes. This reduces the number of bytes used to represent the number. The highest bit of each byte in the varint has a special meaning, if the bit is 1, the subsequent byte is also part of the number, and if the bit is 0, the end. The other 7 bits are used to represent numbers. Therefore, a number less than 128 can be represented by a byte. A number greater than 128 will use two bytes.

See my Last Post

The biggest difference is that the message header is not a fixed length (it is common to use an int 4 bytes fixed length), Varint it uses one or more bytes to represent a number to determine that it is not a fixed length!

The main methods of the Protobufvarint32lengthfieldprepender class are as follows:

@Overrideprotected voidencode (Channelhandlercontext ctx, Bytebuf msg, bytebuf out)throwsException {intBodylen =msg.readablebytes (); intHeaderlen =codedoutputstream.computerawvarint32size (Bodylen); Out.ensurewritable (Headerlen+Bodylen); Codedoutputstream Headerout=Codedoutputstream.newinstance (NewBytebufoutputstream (out), Headerlen);        Headerout.writerawvarint32 (Bodylen);        Headerout.flush ();    Out.writebytes (MSG, Msg.readerindex (), Bodylen); }

Codedoutputstream is mainly for varints related operation classes. First look at how to write the message header, get the Bodylen message body length and then call Computerawvarint32size () to calculate how many bytes are required,

 Public Static intComputerawvarint32size (Final intvalue) {    if(Value & (0xFFFFFFFF << 7)) = = 0)return1; if(Value & (0xFFFFFFFF << 14)) = = 0)return2; if(Value & (0xFFFFFFFF << 21)) = = 0)return3; if(Value & (0xFFFFFFFF << 28)) = = 0)return4; return5; }

0xFFFFFFFF << 72 binary represents 11111111111111111111111110000000, when with value & calculated = 0 means that value is only 000000000000000000000001111111 and one byte is sufficient.

The & operation shows how many bytes are used to represent the current number. The left 7 bits are related to the Varint definition, and the first bit needs to be reserved for the identity (1 indicates that subsequent byte is also part of the number and 0 ends). To represent the first identity bit of the int 32-bit and extra bytes for each byte, 4 bits more, so there will be a maximum of 5 bytes.

Get the Varints value and then how to write out? Then look at the key method WriteRawVarint32 ().

 Public voidWriteRawVarint32 (intValuethrowsIOException { while(true) {      //0x7f to 127      if(value & ~0x7f) = = 0) {//is less than 127, less than one byte can represent theWriterawbyte (value); return; } Else{writerawbyte (value& 0x7F) | 0x80);//because it is not a small 127, plus a high markValue >>>= 7;//Move right 7 bits, then recursion      }    }  }    /**Write a single byte.*/   Public voidWriterawbyte (Final byteValuethrowsIOException {if(Position = =limit)    {Refreshbuffer (); } buffer[position++] =value; }    Private voidRefreshbuffer ()throwsIOException {if(Output = =NULL) {      //We ' re writing to a single buffer.      Throw Newoutofspaceexception (); }    //Since We have a output stream, this is our buffer//and buffer offset = = 0Output.write (buffer, 0, position); Position= 0; }

The value of Byte ( -128~127), 0x7f is 127, 0x80 is 128

The loop takes after 7 bits, if less than 127 ends, not less than the first digit plus the identity bit 1. Because the loop is shifted to the right, the actual position is reversed, the decoding needs to be reversed and then spliced.

Message header because it is varint32 variable byte, so more complex, the message body simple direct writebytes can.

Second, Protobufvarint32framedecoder decoding class

The same corresponds to Codedoutputstream has Codedinputstream class, operation decoding varints.

@Overrideprotected voidDecode (Channelhandlercontext CTX, bytebuf in, list<object> out)throwsException {in.markreaderindex (); Final byte[] buf =New byte[5];  for(inti = 0; i < buf.length; i + +) {            if(!in.isreadable ())                {In.resetreaderindex (); return; } Buf[i]=In.readbyte (); if(Buf[i] >= 0) {                intLength = codedinputstream.newinstance (buf, 0, i + 1). ReadRawVarint32 (); if(Length < 0) {                    Throw NewCorruptedframeexception ("Negative length:" +length); }                if(In.readablebytes () <length)                    {In.resetreaderindex (); return; } Else{out.add (in.readbytes (length)); return; }            }        }        //couldn ' t find the byte whose MSB is off.        Throw NewCorruptedframeexception ("Length wider than 32-bit"); }

The maximum length is 5 bytes, so the 5-length bytes are declared here to read the message header.

Buf[i] >= 0 Why is this here >0 and then you can decode it?

Or this sentence: varints first indicates whether the subsequent byte is part of the number!

If the first bit of the byte is 1, then the subsequent byte is the message header, and when the first bit of the byte is 1, the byte must be negative (the highest bit of the byte is positive or negative), and a number greater than or equal to 0 means that the figure describing the body length of the message has been read.

Then call ReadRawVarint32 () to revert to int, which is the opposite of the previous writeRawVarint32 () line.

 Public intReadRawVarint32 ()throwsIOException {byteTMP =Readrawbyte (); if(TMP >= 0) {     returntmp; }   intresult = tmp & 0x7f; if(TMP = Readrawbyte ()) >= 0) {result|= tmp << 7; } Else{result|= (tmp & 0x7f) << 7; if(TMP = Readrawbyte ()) >= 0) {result|= tmp << 14; } Else{result|= (tmp & 0x7f) << 14; if(TMP = Readrawbyte ()) >= 0) {result|= tmp << 21; } Else{result|= (tmp & 0x7f) << 21; Result|= (tmp = Readrawbyte ()) << 28; if(TMP < 0) {           //Discard Upper.            for(inti = 0; I < 5; i++) {             if(Readrawbyte () >= 0) {               returnresult; }           }           ThrowInvalidprotocolbufferexception.malformedvarint (); }       }     }   }   returnresult;}

Take the nth byte to the left 7*n bit or | The first byte concatenation, realizes the reverse stitching, finally obtains the message body length. The data is then read based on the resulting message body length, and if the message body length is not enough, it is rolled back to Markreaderindex, waiting for the data.

Four, summary

Article source Http://www.cnblogs.com/tankaixiong This paper mainly introduces the decoding and packaging of the PROTOBUF protocol in detail in Netty. The emphasis is illustrated on the algorithm representation of the message header varint32. Understand the implementation of Varint32 in the protocol, convenient application in its language docking.

Netty decoding and Packaging research on PROTOBUF Protocol (2)