Analysis of Netty codec frame of "Turn" Netty series

http://www.infoq.com/cn/articles/netty-codec-framework-analyse/

1. Background 1.1. Coding and decoding technology

Often we also get used to calling encoding (Encode) serialization (serialization), which serializes objects into byte arrays for network transmission, data persistence, or other purposes.

Conversely, decoding (Decode)/deserialization (deserialization) restores the byte array read from the network, disk, and so on to the original object (usually a copy of the original object) to facilitate subsequent business logic operations.

When making remote cross-process service calls (such as RPC calls), you need to encode or decode the objects that need to be transmitted over the network using specific codec techniques to complete the remote invocation.

1.2. Commonly used codec frame 1.2.1. Java serialization

It is believed that the first serialization or codec technology that most Java programmers come into contact with is the serialization mechanism provided by the Java default. Java objects that need to be serialized only need to implement the Java.io.Serializable interface and generate the serialization ID, which can be serialized and deserialized by Java.io.ObjectInput and Java.io.ObjectOutput.

Because of its simplicity and low development thresholds, Java serialization is widely used, but because of its many drawbacks, most RPC frameworks do not select it. The main drawbacks of Java serialization are as follows:

1) cannot cross language: is the most deadly problem in Java serialization. For cross-process service invocations, the service provider may be developed in C + + or other languages, and when we need to interact with heterogeneous language processes, Java serialization is difficult to handle. Because Java serialization technology is a private protocol within the Java language, other languages do not support it, which is completely black-box for the user. The Java serialization of the byte array, the other language can not be deserialized, which seriously hindered its scope of application;

2) The serialized stream is too large: for example, using binary codec technology to encode the same complex Pojo object, its code stream is only about 20% after the Java serialization, the current mainstream codec framework, after the serialization of the code stream is much smaller than the native Java serialization;

3) Poor serialization efficiency: in the same hardware conditions, the same Pojo object to do 100W serialization, binary encoding and Java native serialization performance comparison test as shown: Java native serialization time is 16.2 times times the binary code, the efficiency is very poor.

Figure 1-1 Comparison of binary encoding and Java native serialization performance

1.2.2. Google's Protobuf

PROTOBUF full name Google Protocol buffers, it by Google Open source, in Google's internal and tried. It describes the data structure as a. proto file, through which the code generation tool can generate Pojo objects and PROTOBUF related methods and properties of the corresponding data structure.

It features the following:

1) structured data storage format (Xml,json, etc.);

2) high-efficiency codec and decoding performance;

3) language-independent, platform-independent, good extensibility;

4) Official support for Java, C + + and Python in three languages.

First we look at why XML is not used, although the readability and extensibility of XML is very good, it is well suited to describe the data structure, but the time overhead of XML parsing and the cost of XML for readability are very large, so it is not suitable for high-performance communication protocols. The PROTOBUF uses binary encoding, which provides greater space and performance advantages.

Protobuf Another attractive place is its data description file and code generation mechanism, the advantages of using a data description file to illustrate the structure of the data are as follows:

1) Text data structure Description language, can realize language and platform-independent, especially suitable for the integration between heterogeneous systems;

2) The forward compatibility of the protocol can be achieved by identifying the order of the fields;

3) Automatic code generation, do not need to manually write the same data structure of C + + and Java version;

4) Facilitate the follow-up management and maintenance. Structured documents are easier to manage and maintain than code.

1.2.3. Apache's Thrift

Thrift from Facebook, and in 2007 Facebook submitted thrift as an open source project to the Apache Foundation. For Facebook at the time, the thrift was created to address the large volume of traffic between Facebook's systems and the cross-platform nature of the different locales between systems, so thrift could support a variety of programming languages such as C + +, C #, Cocoa, Erlang , Haskell, Java, Ocami, Perl, PHP, Python, Ruby, and Smalltalk.

Communication between many different languages, thrift can be used as a high-performance communication middleware that supports data (object) serialization and multiple types of RPC services. Thrift is suitable for static data exchange, need to determine its data structure, when the data structure changes, you must re-edit the IDL file, generate code and compile, this is compared with other IDL tools can be regarded as thrift weaknesses. Thrift is suitable for large-scale data exchange and storage of common tools, for large systems of internal data transmission, compared to JSON and XML in the performance and the size of the transmission has a clear advantage.

Thrift mainly consists of 5 parts:

1) language system and IDL compiler: Responsible for generating the interface code of the corresponding language by the user given IDL file;

2) TPROTOCOL:RPC protocol layer, you can choose a variety of different object serialization methods, such as JSON and binary;

3) TTRANSPORT:RPC transmission layer, can also choose different transport layer implementation, such as sockets, NIO, Memorybuffer, etc.

4) Tprocessor: As a link between the protocol layer and the Service implementation provided by the user, the interface responsible for invoking the service implementation;

5) Tserver: Aggregates Tprotocol, Ttransport, and Tprocessor objects.

Our focus is on the codec framework, which corresponds to Tprotocol. Because of the thrift RPC Service invocation and the codec framework binding, the RPC framework is usually used when we use thrift. However, its Tprotocol codec framework can be used independently of the class library.

Similar to PROTOBUF, thrift describes interfaces and data structure definitions through IDL, which supports 8 Java primitives, maps, sets, and lists, supports optional and required definitions, and is very powerful. Because the order of the fields in the data structure can be defined, it can also support the forward compatibility of the protocol.

The thrift supports three more typical codec types:

1) the universal binary codec;

2) compression binary codec;

3) Optimized optional field compression codec.

Due to the support of binary compression codec, thrift codec performance is also very good, far more than Java serialization and RMI and so on.

1.2.4. JBoss marshalling

JBoss marshalling is a serialization API package for Java objects that fixes many issues with the JDK's own serialization package. However, it is compatible with the Java.io.Serializable interface, with some tunable parameters and additional features added, and these parameters and features can be configured through the factory class.

Compared to the traditional Java serialization mechanism, it has the following advantages:

1) Pluggable Class parser provides a more convenient class loading customization strategy, which can be customized through an interface;

2) Pluggable object substitution technique, which does not need to be inherited;

3) Pluggable predefined class cache table, can reduce the length of serialized byte array, promote the object serialization performance of common types;

4) Java serialization can be realized without the implementation of the Java.io.Serializable interface;

5) Improve the serialization performance of objects with caching techniques.

Compared to the two codec frameworks described earlier, JBoss marshalling is more internally used within JBoss and has a limited range of applications.

1.2.5. Other encoding and decoding frameworks

In addition to the above-described codec framework and technology, more commonly used are messagepack, Kryo, Hession and JSON. Limited to the space limit, no longer one by one enumeration, interested friends can self-access relevant information learning.

2. Netty Codec Frame 2.1. Netty Why to provide a codec frame

As a high-performance asynchronous and NIO communication framework, the codec framework is an important part of Netty. Although the codec framework is not part of the Netty micro-kernel, it is essential to extend the codec framework through Channelhandler customization, even though it is in the micro-kernel perspective.

Below we discuss this topic from several angles in detail, first of all, look at the logical architecture diagram of Netty:

Figure 2-1 Netty Logical architecture Diagram

Inbound messages that are read from the network need to be decoded to convert the binary datagram into an application-level protocol message or a business message, which can be identified and processed by the application logic of the upper layer; In the same vein, the user sends the outbound business message to the network, The encoding needs to be encoded into a binary byte array (for Netty is bytebuf) to be sent to the network peer. The encoding and decoding feature is an integral part of the NIO framework, which is essential, whether it is implemented by a business customization extension, or the NIO framework's built-in codec capability.

In order to reduce the user's development difficulty, Netty has decorated the common functions and APIs to shield the underlying implementation details. The customization of the codec function is not very difficult for developers who are familiar with the netty underlying implementation, and are directly based on the Channelhandler extension. But for most beginners or those who are unwilling to understand the underlying implementation details, they need to provide them with simpler libraries and APIs, rather than Channelhandler.

Netty is doing very well in this respect, it provides a universal codec framework for users to extend, and also provides a common codec class library for users to use directly. On the basis of ensuring custom extensibility, we try to reduce the user's development workload and development threshold and improve the development efficiency.

The list of codec functions for Netty presets is as follows: Base64, Protobuf, JBoss marshalling, spdy, etc.

Figure 2-2 List of codec functions for Netty presets

2.2. Commonly used decoder 2.2.1. Linebasedframedecoder Decoder

Linebasedframedecoder is a carriage return newline decoder that, if a user sends a message with a carriage return newline as the identity of the end of the message, the message can be decoded directly using the Netty Linebasedframedecoder. You only need to add linebasedframedecoder correctly to Channelpipeline when initializing the Netty server or client, and you do not need to re-implement a set of line-wrapping decoders yourself.

Linebasedframedecoder works by iterating through the readable bytes in the bytebuf in turn, judging if there is a "\ n" or "\ r \ n", and, if so, where it ends, a row of bytes from the readable index to the end position interval. It is a decoder that ends with a newline character, supports either a carry terminator or no terminator, and supports the maximum length of the configuration line. If a newline character is not found after continuous reading to the maximum length, an exception is thrown, ignoring the previously read exception stream. Prevents a system memory overflow because the datagram does not carry a newline character resulting in an unrestricted backlog of bytebuf received.

It uses the following effects:

Before decoding: +------------------------------------------------------------------+                        received datagram "This was a netty example for Using the NIO framework.\r\n when you "+------------------------------------------------------------------+ After decoding the Channelhandler received the following object: +------------------------------------------------------------------+                        The text message after decoding "This was a netty example for using the NIO framework." +------------------------------------------------------------------+

Usually, Linebasedframedecoder will be used in conjunction with Stringdecoder, combined into a line-switched text decoder, text-class protocol parsing, text wrapping decoder is very useful, such as the HTTP message header parsing, FTP protocol message parsing and so on.

Here's a simple example of using a text wrap decoder:

@Overrideprotected void Initchannel (Socketchannel arg0) throws Exception {   arg0.pipeline (). AddLast (New Linebasedframedecoder (1024x768));   Arg0.pipeline (). AddLast (New Stringdecoder ());   Arg0.pipeline (). AddLast (New Userserverhandler ());

To initialize the channel, first add the Linebasedframedecoder to Channelpipeline, and then add the string decoder Stringdecoder in turn, and the business handler.

2.2.2. Delimiterbasedframedecoder Decoder

Delimiterbasedframedecoder is a delimiter decoder that allows a user to specify the end of a message delimiter, which can be automatically decoded as a delimiter as the end of a stream is identified by the message. The carriage return line-wrap decoder is actually a special Delimiterbasedframedecoder decoder.

Delimiter decoder in the actual work also has a wide range of applications, the author of the telecommunications industry, a lot of simple text private protocol, is a special delimiter as the message end of the identity, especially for those who use long connections text-based private protocol.

Delimiter designation: Unlike everyone's habits, the delimiter is not a char or a string as a construction parameter, but rather a bytebuf, so let's take a practical example to illustrate its usage.

If the message is "$_" as a delimiter, the server-side or client-initiated code instance of Channelpipeline is as follows:

@Overridepublic void Initchannel (Socketchannel ch) throws Exception {    Bytebuf delimiter = Unpooled.copiedbuffer ("$_ "    . GetBytes ());   Ch.pipeline (). AddLast (    new Delimiterbasedframedecoder (1024x768,    delimiter));   Ch.pipeline (). AddLast (New Stringdecoder ());   Ch.pipeline (). AddLast (New Userserverhandler ());

First convert "$_" to Bytebuf object, construct Delimiterbasedframedecoder as a parameter, add it to channelpipeline, then add a string decoder (usually for text decoding) and user handler, Note that the order in which the decoders and handler are added, if reversed, causes the message decoding to fail.

Delimiterbasedframedecoder principle Analysis: When decoding, determine whether the current read bytebuf contains delimiter bytebuf, if included, then intercept the corresponding bytebuf return, the source code is as follows:

Detailed analysis of the implementation of the indexof (buffer, Delim) method, the code is as follows:

Similar to the search algorithm in Java string, the algorithm uses two pointers for the original string to search, and returns the index position if the search succeeds, otherwise returns-1.

2.2.3. Fixedlengthframedecoder Decoder

Fixedlengthframedecoder is a fixed-length decoder that can automatically decode messages at a specified length, and developers do not need to consider TCP's sticky/unpacking issues.

For fixed-length messages, if the actual length of the message is less than the fixed length, it tends to perform a complement operation, which in some way leads to a waste of space and resources. However, its advantages are very obvious, the codec is relatively simple, so in the actual project still have a certain application scenario.

With the Fixedlengthframedecoder decoder, no matter how many datagrams are received at a time, it is decoded according to the fixed length set in the constructor, if it is a half-packet message, The Fixedlengthframedecoder caches the half-packet message and waits for the next packet to arrive after the package, until it reads a full package.

If the length of a single message is 20 bytes, the effect of using the Fixedlengthframedecoder decoder is as follows:

Before decoding: +------------------------------------------------------------------+ incoming                        datagram "HELLO NETTY for USER DEVELOPER "+------------------------------------------------------------------+ decoded: +--------------------------- ---------------------------------------+                        decoded datagram "HELLO NETTY for USER" +-------------------------------------- ----------------------------+

2.2.4. Lengthfieldbasedframedecoder Decoder

Readers who understand the TCP communication mechanism should be aware of the sticky and unpacking of the TCP underlying, and when we receive the message, the messages that are not considered to be read are an entire packet, especially for non-blocking I/O and long-connection communication.

How to differentiate an entire packet of messages, usually with the following 4 practices:

1) fixed length, for example every 120 bytes represents an entire packet of messages, insufficient front complement. The decoder is relatively simple when dealing with this kind of constant message, and is decoded after each reading of a specified length of word;

2) Differentiate messages by carriage return newline character, such as HTTP protocol. This kind of message-distinguishing method is used for text protocol.

3) distinguish the whole packet message by a specific delimiter;

4) Identify the entire package message by setting the Length field in the protocol header/message header.

The previous chapters of the first three decoders have been described in detail, so let's learn about the last common decoder-lengthfieldbasedframedecoder.

Most protocols (private or public), the protocol header carries a length field that identifies the body of the message or the length of the whole packet message, such as SMPP, HTTP protocol, and so on. Because of the universality based on the length decoding requirement, and in order to reduce the difficulty of the user's protocol development, Netty provides the lengthfieldbasedframedecoder, automatically shielding the TCP bottom of the unpacking and sticking problem, only need to pass in the correct parameters, can easily solve the "read half package" problem.

Let's look at how to implement different "half-packet" read strategies with different combinations of parameters. The first common way is that the first field of the message is the length field, followed by the message body, and the message header contains only one length field. Its message structure is defined:

Figure 2-3 Byte buffers before decoding (14 bytes)

Use the following combination of parameters to decode:

1) lengthfieldoffset = 0;

2) Lengthfieldlength = 2;

3) lengthadjustment = 0;

4) Initialbytestostrip = 0.

Decoded byte buffer contents:

Figure 2-4 Decoded byte buffers (14 bytes)

With the Bytebuf.readablebytes () method we can get the length of the current message, so the decoded byte buffer can not carry the length field, because the length field is at the starting position and the length is 2, so Initialbytestostrip is set to 2, The parameter combination is modified to:

1) lengthfieldoffset = 0;

2) Lengthfieldlength = 2;

3) lengthadjustment = 0;

4) Initialbytestostrip = 2.

Decoded byte buffer contents:

Figure 2-5 Skipping the length field decoded byte buffers (12 bytes)

The decoded byte buffer discards the length field, contains only the message body, and for most protocols, the message length is not useful after decoding, so it can be discarded.

In most scenarios, the Length field is used only to identify the length of the message body, which typically consists of the message length field + message body, as shown in a few examples. However, for some protocols, the length field also contains the length of the message header. In this scenario, it is often necessary to use lengthadjustment for remediation. Because the length of the entire message (including the message header) is often greater than the length of the message body, the lengthadjustment is negative. Figure 2-6 shows the length of the message header by specifying the Lengthadjustment field:

1) lengthfieldoffset = 0;

2) Lengthfieldlength = 2;

3) Lengthadjustment =-2;

4) Initialbytestostrip = 0.

Before decoding the code stream:

Figure 2-6 contains the code flow for the length field itself

Code stream after decoding:

Figure 2-7 decoding the code stream

Because of the wide variety of protocols, not all protocols place the length field at the top of the message header, and when the field that identifies the length of the message is in the middle or tail of the message header, it needs to be identified by using the Lengthfieldoffset. The following combination of parameters shows how to resolve a problem where the message length field is not in the first place:

1) Lengthfieldoffset = 2;

2) Lengthfieldlength = 3;

3) lengthadjustment = 0;

4) Initialbytestostrip = 0.

Where Lengthfieldoffset represents the number of bytes that the length field offsets in the message header, Lengthfieldlength represents the length of the length field itself, and the decoding effect is as follows:

Before decoding:

Figure 2-8 Source stream for length field offset

After decoding:

Figure 2-9 The code stream after the length field offset decoding

Because the length of the message header 1 is 2, the offset of the length field is 2, and the message length field is length 3, so the Lengthfieldlength value is 3. Because the Length field only identifies the length of the message body, both Lengthadjustment and Initialbytestostrip are 0.

The last scenario is that the length field is clamped between the two message headers or the length fields in the middle of the message header, and there are other message header fields before and after, in which case if you want to ignore the length field and the other message header fields that precede it, The Initialbytestostrip parameter can be used to skip the length of bytes to be ignored, and its combination configuration is as follows:

1) lengthfieldoffset = 1;

2) Lengthfieldlength = 2;

3) lengthadjustment = 1;

4) Initialbytestostrip = 3.

Code stream before decoding (16 bytes):

Figure 2-10 The original stream (16 bytes) of the length field sandwiched in the middle of the message header

Code stream after decoding (13 bytes):

Figure 2-11 Code stream after decoding the length field clip in the middle of the message header (13 bytes)

Because the length of the HDR1 is 1, the offset of the length field is Lengthfieldoffset 1 and the Length field is 2 bytes, so lengthfieldlength is 2. Since the length field is the length of the message body, if the field in the message header is carried after decoding, it needs to be adjusted with Lengthadjustment, where its value is 1, which represents the length of the HDR2, and finally because the decoded buffer ignores the length field and the HDR1 part. So Lengthadjustment is 3. The decoded result is 13 bytes, and the HDR1 and length fields are ignored.

In fact, through the 4 parameters of different combinations, can achieve different decoding effect, users in the process can be used in accordance with the actual situation of the business to adjust flexibly.

Because of the sticky and packet problems with TCP, it is common for users to handle half-pack messages themselves. Using the Lengthfieldbasedframedecoder decoder can automatically solve the problem of half-packet, its customary usage is as follows:

Pipeline.addlast ("Framedecoder", New Lengthfieldbasedframedecoder (65536,0,2));p ipeline.addlast ("UserDecoder", new Userdecoder ());

Add the Lengthfieldbasedframedecoder decoder in pipeline, specify the correct combination of parameters, it can decode Netty bytebuf into the whole packet message, the subsequent user decoder gets a complete datagram, Decoding according to logic Normal, no longer need to consider the "read half package" problem, reduce the user's development difficulty.

2.3. Commonly used Encoders

Netty does not provide encoders matching 2.2 chapters for the following reasons:

1) The 4 commonly used decoders in the 2.2 chapter are parsing a complete datagram to the backend, mainly to solve the TCP underlying sticky packet and unpacking, and for encoding, which is to serialize the Pojo object to Bytebuf, does not need to deal with the TCP plane, there is no half-packet encoding problem. From the perspective of application scenarios and practical problems to be solved, the two sides are non-reciprocal;

2) It is difficult to abstract the appropriate encoder, for different user and application scenarios, serialization technology is not the same, the Netty bottom of the unified abstraction package is not appropriate.

Netty provides a rich codec framework for user integration by default, this article explains the more commonly used Java serialization encoders. Other encoders are implemented in a similar way.

2.3.1. Objectencoder Encoder

Objectencoder is a Java serialization encoder that is responsible for serializing an object that implements the serializable interface to byte [] and then writing to BYTEBUF for the cross-network transmission of the message.

Let's analyze its implementation together:

First, we find that it inherits from Messagetobyteencoder, and its role is to encode objects into BYTEBUF:

If you want to use Java serialization, the object must implement the Serializable interface, so its generic type is serializable.

Messagetobyteencoder subclasses only need to implement encode (Channelhandlercontext ctx, I msg, bytebuf Out) method, below we focus on the implementation of the Encode method:

First, Bytebufoutputstream and ObjectOutputStream are created to serialize object objects into Bytebuf, and it is worth noting that the Length field (4 bytes) must be reserved before writeobject. The update for the subsequent length field.

Writes the length placeholder (4 bytes), followed by the serialized object object, then calculates the length of the stream after serialization based on the Writeindex of Bytebuf, and finally calls the setint of bytebuf (int index, int value) Update the length placeholder for the actual bitstream length.

One detail to note is that the update stream length field uses the Setint method instead of Writeint, because the Setint method only updates the content and does not modify Readerindex and Writerindex.

3. Netty Codec Framework Customizable

Although Netty has a rich set of codec class libraries in place, it is always necessary to customize the codec function in the actual business development process. With the Netty codec framework, protocol customization can be very convenient. This section will explain the commonly used support custom codec class library, in order to enable readers to familiarize and master the codec framework as soon as possible.

3.1. Decoder 3.1.1. Bytetomessagedecoder Abstract Decoder

When using NIO for network programming, it is often necessary to decode a read byte array or byte buffer into a Pojo object that the business can use. In order to facilitate the business to decode Bytebuf into a business Pojo object, Netty provides the Bytetomessagedecoder abstract tool decoding class.

User-defined decoder inherits Bytetomessagedecoder, only needs to implement void Decode (Channelhandler Context ctx, bytebuf in, list<object> Out) abstract method to complete decoding of bytebuf to Pojo objects.

Since Bytetomessagedecoder does not consider scenarios such as TCP sticky and unpacking, the user-defined decoder needs to handle the "read half-packet" problem on its own. Because of this, most scenarios do not inherit bytetomessagedecoder directly, but instead inherit some of the more advanced decoders to mask half-packet processing.

In a real project, Lengthfieldbasedframedecoder and Bytetomessagedecoder are typically used in combination to decode the network read datagram into an entire packet message, which decodes the entire packet message into the final business object.

In addition to forming new decoders in combination with other decoders, Bytetomessagedecoder is also the parent of many base decoders, as shown in the inheritance relationship:

Figure 3-1 Bytetomessagedecoder inheritance diagram

3.1.2. Messagetomessagedecoder Abstract Decoder

Messagetomessagedecoder is actually a netty two-time decoder whose duty is to decode an object two times into other objects.

Why call it a two-time decoder? We know that the TCP datagram read from Socketchannel is Bytebuffer, which is actually a byte array. We first need to read out the datagram in the Bytebuffer buffer and decode it as a Java object, then decode the Java object two times according to some rules and decode it to another Pojo object. Because Messagetomessagedecoder after Bytetomessagedecoder, so called two times decoder.

Two times the decoder is very useful in real business projects, in the case of the Http+xml protocol stack, the first decoding is often to decode the byte array into a HttpRequest object, and then decode the message body string in the HttpRequest message two times. Decodes an XML-formatted string into a Pojo object, which uses a two-time decoder. There are many more scenarios like this, no longer one by one enumerations.

In fact, making a super-complex decoder that combines multiple decoders into a chatty Messagetomessagedecoder decoder also seems to solve multiple decoding problems, but code maintainability in this way can be very poor. For example, if we are going to add a print stream to the Http+xml protocol stack, we will print the XML-formatted stream after the first decoding of the acquired HttpRequest object. If the combination of multiple decoders, in the middle of a print the body of the handler can be inserted, do not need to modify the original code, if you do a chatty decoder, you need to decode the method to increase the code stream, scalability and maintainability will become worse.

The user's decoder only needs to implement the void decode (Channelhandlercontext ctx, I msg, list<object> Out) abstract method, because it is to decode one pojo to another pojo, So it is generally not involved in half-package processing, which is simpler than bytetomessagedecoder. Its inheritance diagram is as follows:

Figure 3-2 Messagetomessagedecoder Decoder inheritance diagram

3.2. Encoder 3.2.1. Messagetobyteencoder Abstract Encoder

Messagetobyteencoder is responsible for encoding the Pojo object into Bytebuf, the user's encoder inherits the message Tobyteencoder, implements void Encode (Channelhandlercontext ctx, I MSG, bytebuf out) interface interface, the sample code is as follows:

public class Integerencoder extends messagetobyteencoder<integer> {      @Override public      void Encode ( Channelhandlercontext CTX, Integer msg,bytebuf out)         throws Exception {             out.writeint (msg);          }      }

Its implementation principle is as follows: When the write operation is called, first determine whether the current encoder supports the message that needs to be sent, if it is not supported then direct transmission, if it is supported to determine the type of buffer, for direct memory allocation Iobuffer (out of heap memory), for heap memory through the Heapbuffer method allocation , the source code is as follows:

After the buffer allocation used by the encoding is complete, call the Encode abstract method to encode the method defined as follows: it is responsible for the implementation of subclasses.

After the encoding is complete, call the release method of Referencecountutil to free the encoded object MSG. The following judgment is made on the encoded BYTEBUF:

1) If the buffer contains a byte that can be sent, call Channelhandlercontext's Write method to send the BYTEBUF;

2) If the buffer does not contain writable bytes, it is necessary to release the encoded BYTEBUF and write an empty bytebuf to Channelhandlercontext.

After the send operation completes, the encoding buffer Bytebuf object is released before the method exits.

3.2.2. Messagetomessageencoder Abstract Encoder

Encode a Pojo object into another object, taking the Http+xml protocol as an example, by first encoding the Pojo object into an XML string, and then encoding the string as an HTTP request or a reply message. For complex protocols, it is often necessary to undergo multiple coding, in order to facilitate the expansion of functions, can be implemented through a plurality of encoder combinations to achieve the relevant functions.

The user's decoder inherits the Messagetomessageencoder decoder and implements the void encode (Channel handlercontext ctx, I msg, list<object> out) method. Note that the difference between it and Messagetobyteencoder is that the output is a list of objects instead of BYTEBUF, the sample code is as follows:

public class Integertostringencoder extends Messagetomessageencoder <Integer>  {          @Override          public void encode (Channelhandlercontext ctx, Integer message,              list<object> out)                  throws Exception           {              Out.add (Message.tostring ());          }      }

The Messagetomessageencoder encoder's implementation principle is similar to the previous analysis of the Messagetobyteencoder, the only difference is that its encoded output is an intermediate object, not the final transportable bytebuf.

Simply look at its source code implementation: Create a Recyclablearraylist object, determine whether the currently-encoded object is a type that the encoder can handle, and if not, ignore it and execute the next Channelhandler write method.

The specific encoding method is implemented by the user sub-class encoder, if the encoded recyclablearraylist is empty, the encoding is not successful, releasing the Recyclablearraylist reference.

If the encoding succeeds, the encoded Pojo object is iterated by traversing recyclablearraylist, and the code is as follows:

3.2.3. Lengthfieldprepender Encoder

If the first field in the protocol is a length field, Netty provides a Lengthfieldprepender encoder that calculates the length of the binary bytes of the current message to be sent and adds that length to the BYTEBUF buffer header:

Figure 3-3 Lengthfieldprepender Encoder

By Lengthfieldprepender, the length of the message to be sent is written to the first 2 bytes of the BYTEBUF, and the encoded message consists of the length field + the original message.

By setting Lengthfieldprepender to True, the message length will contain the number of bytes occupied by the length itself, and after opening lengthfieldprepender, the encoding results in the example shown in Figure 3-3 are as follows:

Figure 3-4 Encoding effect after opening the lengthfieldprepender switch

Lengthfieldprepender working principle analysis is as follows: first set the Length field, if you need to include the message length itself, then add the length of the lengthfieldlength on the basis of the original length.

If the adjusted message length is less than 0, the argument is thrown with an illegal exception. Determine the number of bytes that the message length itself occupies in order to write the length field to bytebuf in the correct way, with the following 6 possibilities:

1) The Length field occupies 1 bytes: If 1 byte bytes are used to represent the message length, the maximum length is less than 256 bytes. Check the length, if the checksum fails, throw the parameter illegal exception, if the validation passes, create a new bytebuf and write the length value to Bytebuf by WriteByte;

2) The Length field occupies 2 bytes: If 2 byte bytes are used to represent the message length, the maximum length needs to be less than 65,536 bytes, the length is checked, and if the checksum fails, the argument is thrown illegally; A new bytebuf is created and the length value is written to Bytebuf by Writeshort;

3) The Length field occupies 3 bytes: If 3 byte bytes are used to represent the message length, the maximum length needs to be less than 16,777,216 bytes, the length is checked, and if the checksum fails, the argument is thrown illegally; A new bytebuf is created and the length value is written to Bytebuf by Writemedium;

4) The Length field occupies 4 bytes: A new bytebuf is created and the length value is written to Bytebuf by Writeint;

5) The Length field occupies 8 bytes: A new bytebuf is created and the length value is written to Bytebuf by Writelong;

6) Other length values: Throw error directly.

The relevant code is as follows:

Finally, the original need to send the bytebuf copied to list<object> out, complete the encoding:

4. Introduction of the author

Li Linfeng, who graduated from Tohoku University in 2007 and entered  in 2008 for the design and development of high performance communication software, has 7 years of experience in NIO design and development, is proficient in Netty, Mina and other NIO frameworks and platform middleware, and is currently the architect of  Software platform architecture, the Netty authoritative guide Author

Contact information: Sina Weibo nettying:nettying public number: Netty House

For Netty learning problems, or think of valuable netty or NIO related cases, you can contact me through the above several ways.

Analysis of Netty codec frame of "Turn" Netty series