Netty in Action (20) Chapter codecs

Tenth chapter, Chapter 11th preface

For the network, the data is just the raw byte sequence, but our program organizes these bytes in a way that we can understand, and we generally call this information "information", Convert the information into bytes or replace the bytes in the network with the information we can read these are among the most common tasks in our network transmission, and you may need to work in a standard format or protocol, such as an FTP protocol or a Telnet protocol, or a proprietary protocol that is customized from a third party. Or to inherit an existing information format according to the application of the word self

A component that handles the conversion of data from a network into application data is called a decoder or encoder, and, in contrast, a separate component, the two functions of which we call this component a decoder, Netty provides a number of tools to create these decoders, We can do this by refactoring some of the classes to achieve some of the more famous common transport decoders such as HTTP,BASE64, and so on, to meet your custom needs in this way

The tenth chapter is the basic introduction of coding and decoding, you will learn some classic user stories to understand the netty most basic decoder, then you will learn how to apply these classes to the entire Netty framework, You will find that these decoders can be built on the API of the components we have previously learned, so you can quickly get started using these classes

In the 11th chapter, we will explore how the decoders and encoders provided by Netty work in some more specific scenarios, and WebSocket is a more interesting section, so we're going to discuss this advanced network transport protocol in detail in the third module.

In the tenth chapter, the contents of this chapter include:

1) A basic explanation of the encoder, decoder, decoder

2) Netty Coding Tool Class

Since many professional frameworks provide many standard architectural design patterns, some of the more common data processing patterns are often used as the hottest candidates for combat applications, and these more commonly used solutions can save developers a lot of time and experience

The subject matter of this chapter must be coding or decoding, or converting data from one format to another, the components that deal with this problem, which we often call decoders, Netty provides components for many protocols to simplify the implementation of custom decoders, for example, You will find that the Netty decoder can seamlessly support the implementation of the POP3,IMAP,SMTP protocol.

10.1 What is a codec?

Each application framework needs to define how to convert raw bytes or parse raw bytes at both ends of the remote transmission, consider how to turn the target information into raw bytes or reverse the process to convert the raw bytes into the target programming information, the logic of the conversion is handled by the decoder, it contains the function of encoding and decoding, Each decoder can convert a string of bytes from one format to another, so how do we differentiate between encoding and decoding?

Imagine a scenario in which a message is used in our application in a structured sequence of bytes, so that the encoder translates this information into a format suitable for transmission in the network, such as a byte stream, which corresponds to a decoder, converting the stream of words returned from the network to the information format that the program can recognize. In short, the encoder operates an output data, the decoder operates an input data

Keep this knowledge background in mind and let's see how Netty is going to implement both of these components

10.2 Decoders

In this section, we will look into the Netty decoder and use a few specific cases to illustrate when it is appropriate to work with these classes, which are divided into two different user cases:

1) Decode bytes into information-----bytetomessagedecoder and Replayingdecoder
2) Convert one information format into another----------Messagetomessagedecoder

Because the decoder is responsible for converting the transferred input data into another format, you will not be surprised if you know that the Netty decoder implements Channelinboundhandler.

So when can you use a decoder? Very simple: When you need to transfer the input data to the next Channelinboundhandler in Channelpipeline, there is the design that we need to thank channelpipeline, You can use multiple decoders in a chained format to connect all the formats together to handle any complex transport logic, the main purpose of which is to Netty support modularity and re-use

10.2.1 Abstract class Bytetomessagedecoder

Converting bytes into information is a common task for using the abstract class Bytetomessagedecoder provided by Netty, since you cannot know whether the remote data input can transmit the entire information at once, This class will buffer all input data until the input bytes satisfy the processing requirements, and table 10.1 shows you the two most important methods of this class.

We give an example of how to use this class, assuming you need to receive a byte stream, which contains only simple int variables, each int variable can be handled separately, in this case, Each time you read an int variable in the input bytebuf, it is transferred to the next Channelinboundhandler in the pipeline, in order to decode the byte stream in the bytebuf, we need to inherit Bytetomessagedecoder

Our design 10.1 shows the same:

Each time you read four bytes from the input bytebuf, decode it into an int variable, and then add the decoded int variable to the list, and when no more objects are added to the list, Then the contents of the list will be sent to the next channelinboundhandler.

The following code listing shows the code for the Tointegerdecoder

Although Bytetomessagedecoder can make this conversion mode very simple, you may find that every time you need to verify that there are enough bytes in the input bytebuf to convert to int, the number is verified. The next section we will discuss Replayingdecoder, a special decoder, this decoder can eliminate this step at a very small cost

TIPS: Technical references in decoders:

As we have mentioned in the fifth and sixth chapters, there are some important things that need to be worth in counting applications, in the case of decoders and decoders, the process is simple, and once the information has been decoded or encoded, it will automatically call Referencecountutil.release (message) method to release itself, if you need to continue to get references to these objects for later use, you can call the Referencecountutil.retain (message) method to get a continuous reference to this object because the value of the count reference increases, which prevents the information from being released

10.2.2 Abstract class Replayingdecoder

Replayingdecoder inherited the Bytetomessagedecoder so that we can free ourselves from the method of calling Readablebytes, It accomplishes this requirement by wrapping the input bytebuf with a custom BYTEBUF implementation Replayingdecoderbuffer, a complete declaration of the following:

The parameter s specifies the type to use for state management, where void specifies that no value will be specified, and the following code listing re-implements the ToIntegerDecoder2 function with Replayingdecoder

Previously, the variables of type int were evolved from BYTEBUF and then added to the list, and if insufficient bytes were output, then the Readint method would throw an error that would be caught and processed by the underlying class. Then the Decode method will be called again when the output of more bytes is executed.

Here are a few important aspects to note about Replayingdecoder:

1) It does not support all modes of operation of all BYTEBUF, if an unsupported way is called, then a Unsupportedoperationexception exception will be thrown

2) Replayingdecoder execution efficiency is slightly worse than Bytetomessagedecoder

If you compare the code list of 10.1 and 10.2, you will find the latter simpler, of course, the example itself is very simple, but it is important to keep in mind that using bytetomessagedecoder in a real production environment or using Replayingdecoder's choice is actually Will affect the performance of your program, we have a user guidelines here you can refer to: In a very simple business scenario do not want to introduce additional complexity in the case of the use of bytetomessagedecoder, or use Replayingdecoder

TIPS: More Decoders

Here are some classes to handle more complex user requirements

1) Io.netty.handler.codec.LineBasedFrameDecoder This class is used internally by Netty, using the line end control \ n or \ r \ n to parse the information data

2) Io.netty.handler.codec.http.HttpObjectDecoder This decoder is used to process HTTP types of data

You can find some additional decoders and decoders under Io.netty.handler.codec's sub-package to handle some of the more specific user stories, see the Netty Java documentation for details.

10.2.3 Abstract class Messagetomessagedecoder

In this section, we'll show you how to convert between two kinds of information formats, for example, to transform an appropriate pojo into another format pojo, and we use the abstract class of the base class

The parameter I specifies the type of the MSG parameter in the Decode method, Decode is the only method that needs to be implemented, and table 10.2 shows the details of this method

In this example, we write a decoder that inherits the Integertostringdecoder from the messagetomessagedecoder<integer>, The Decode method will convert the integer parameter to string, and the following is the complete declaration of this method:

As before, a string variable of the output of the decoder will be added to the list and then transferred to the next handler in the pipeline.

This design is shown in Figure 10.2:

The following code listing is a concrete implementation of Integertostringdecoder:

For some more complex examples, you can look at the Io.netty.handler.codec.http.HttpObjectAggregator class, which inherits the Messagetomessagedecoder

10.2.4 Class toolongframeexception

Since Netty is an asynchronous framework, there are times when you need to put the number of cache bytes into memory until you can read the decoded bytes, so you can't decode the cache bytes larger than the available memory space, in order to handle this more common concern, Netty provides an exception to the type of toolongframeexception that will be thrown if a frame data exceeds the specified size limit at the time of decoding.

In order to prevent this exception, you can set the maximum byte of the frame data of the valve, if exceeded, will result in throwing toolongframeexception exception, the processing of this exception depends entirely on the decoding end of the user, some scenarios, For example HTTP protocol, may allow you to return a special return, in some other scenarios, there may be only one option is to close the connection

The Code Listing 10.4 shows you how a bytetomessagedecoder can be used to the full toolongframeexception Exception to notify the other Channelhandler in the pipeline that the frame data has exceeded the valve's exception, please note that this protection is important, especially if you work with a protocol that has variable-size frame data in case

So far, we have learned some of the common use of decoders, but also learned some of the netty provided some of the abstract classes to build these classes, but decoding only one side of the decoding coin, the other side of the decoder is the encoder, the encoder can transform the information into a suitable type of transmission, The detailed API for the encoder will be discussed in the next section

10.3 encoders

Recalling our previous definition, an encoder implements the Channeloutboundhandler, converting one format of data into another format for transmission, which has the opposite function with the decoder we have learned, and Netty provides a number of classes to help you write encoders for the following functions

1) encoding from message to byte

2) encoding from message to MESSAG

We will start by learning an abstract basic class Messagetobyteencoder as we learn these encoders

10.3.1 Abstract class Messagetobyteencoder

Before we used Bytetomessagedecoder to convert bytes into a message we could recognize, now we use Messagetobyteencoder to do the opposite, Table 10.3 shows the API for the Messagetobyteencoder

You may have noticed that there is only one method for this class, but the corresponding decoder is made up of two methods, and the reason why the decoder has two methods is because the decoder usually needs to generate the last message when the channel is closed, which obviously does not appear in the encoder. It makes no sense to produce a message after the connection is closed.

Figure 10.3 shows a shorttobyteencoder receives an instance of a short type as information, encodes it into the original short variable, then writes it to BYTEBUF, and finally transmits it to the pipeline in a channeloutboundhandler, each output of sh Ort will occupy 2 bytes in the Bytebuf

The following code listing shows the specific implementation of Shorttobyteencoder:

Netty offers a few professional messagetobyteencoder dependent on these encoder you can build your own implementation, class Websocket08frameencoder provides a good example of the actual combat, You'll be io.netty.handler.codec.http.websocketx under the bag.

10.3.2 Abstract class Messagetomessageencoder

You've learned how to convert an input data from one format to another, and in order to complete the entire blueprint of the decoder, we'll show you how to convert from one format to another for an output data, This implementation is provided by the Messagetomessageencoder encode method, as described in table 10.4

To illustrate this, code listing 10.6 writes a integertostringencoder, which inherits from Messagetomessageencoder, which is shown in the design 10.4:

In the following code listing, an encoder converts an integer into a string and stores it in the list

For more Messagetomessageencoder professional use, you can view Io.netty.handler.codec.protobuf.ProtobufEncoder, This class can handle data formats defined by Google protocol buffer.

10.4 Abstract Codec classes

Although we have learned the decoder and encoder as two different components, there are times when you will find it more efficient to manage data transfer in the same class, and Netty's abstract decoder class can help us to do this function, Because the decoder is bound to the decoder and encoder to handle the two types of operations we have just learned, you may question whether the class of these decoders is not inheriting two classes, One is Channelinboundhandler, one is Channeloutboundhandler.

In the real production environment, why do we always use the encoder or decoder instead of the compatible decoder, because using these two components alone can maximize the reuse and extensibility of the code, which conforms to the Netty design principle

When we are studying the decoder, we will compare and weigh the decoder with the corresponding decoder or encoder.

10.4.1 Abstract class Bytetomessagecodec

Let's start with a case study of the decoder, in this case we need to decode the byte into some kind of information, possibly a pojo, and then encode the Pojo again, Bytetomessagecodec This class can help us to do this function, Because it integrates the Bytetomessagedecoder and the class messagetobyteencoder that are opposite to the Bytetomessagedecoder function, Some of the core methods of the Bytetomessagecodec class are shown in table 10.5

The use of any protocol for the request response class Bytetomessagecodec This class should be a good candidate, for example, in an SMTP implementation, this decoder can read in the bytes entered and then decode it into a custom data type, commonly referred to as " Smtprequest ", at the receiving end, when a response arrives, a corresponding smtpresponse will also be created, which will re-encode the object just decoded into a new object for transmission

10.4.2 Abstract class Messagetomessagecodec

In Chapter 10.2.2 we learn that a class that inherits from Messagetomessageencoder is used to convert a message into another format, and we can use a separate class Messagetomessagecodec to complete this polling process Messagetome Ssagecodec is a class that specifies a parameter, which is defined as follows:

Some of the more important methods of this approach are shown in table 10.6:

The decode method converts a inbound_in type of information into a outbound_in type, and then encode the method to do the opposite, which can help you imagine that the inbound_in type of data can be transferred directly, and then Out_ Bound information can be applied to direct processing applications

While this decoder may seem a bit esoteric, it is relatively simple to use: When you use these two different message APIs to convert information four times, we often encounter this scenario, For example, it is appropriate to use this class when we have to interact with a type of data API.


Tips:websocket Protocol

In the following example we are referring to WebSocket's written Messagetomessagecodec, which is a relatively modern protocol, it can be fully browser-side and server-side two-way communication, We'll discuss Netty's support for websocket in depth in the 11th chapter.


Code Listing 10.7 shows how many conversions occurred in a single communication, and our custom Websocketconverthandler inherited Messagetomessagecodec with websocketframe specified inbound_ The type of the in parameter, with the Mywebsocketframe class specifying the OUTBOUND_IN variable, which is the static inner class of the Websocketconverthandler class itself






10.4.3 Class combinedchannelduplexhandler

As we mentioned before, if a combination of an encoder or a decoder can have a certain impact on our ability to reuse, we have a way to eliminate the problem and we do not need to put an encoder and a decoder in the same unit class, This workaround is provided by Combinedchannelduplexhandler and its complete declaration is as follows:


This class can be seen as a container of Channelinboundhandler and Channeloutboundhandler, which, through the ability to inherit a pair of encoders and decoders, We do not need to directly inherit an abstract decoder to implement our own functions, we use the following a complete example to illustrate the problem:
First, we can take a look at the Bytetochardecoder in the code listing below, noting that this implementation inherits and Bytetomessagedecoder, because this class needs to read the char type data from the BYTEBUF

Here the Decode method extracts 2 bytes from the Bytebuf each time and writes it to the list as a char, noting that this can be automatically boxed into a character object
The following code listing is the Chartobyteencoder class, which then converts char to Byte, which inherits from Messagetobyteencoder because it needs to recode char into bytes, which can be directly written directly by the Bytebuf method.:

Now that we have the encoder and the decoder, let's join it together and build it into a decoder, and the code listing below shows how it can be implemented.

As you can see, this approach may be simpler and more flexible in some scenarios than directly using a decoder, which is, of course, a matter of personal preference in the final analysis.

10.5 Summary

In this section, we learned how to use the API of the Netty decoder, learned how to write encoders and decoders, and you know why using these APIs is better than using Channelhandler APIs directly

You know how an abstract decoder provides the ability to encode and decode in a class, and of course, if you need more flexibility or re-usability, you can also combine the two functions, without having to inherit any abstract decoders.

In the next section, we will discuss the implementation of Channelhandler, explaining more implementation details of the original decoder as part of the Netty framework itself, and using the original decoder to accomplish some special protocols or tasks.

Netty in Action (20) Chapter codecs