Netty Decoder Abstract Parent class Bytetomessagedecoder source parsing

Objective

There are many kinds of decoders for Netty, such as the length-based, delimiter-based, private protocol. However, the overall thinking is consistent.

Unpacking idea: When the data satisfies the decoding condition, it is disassembled. Put it into an array. It is then sent to the business handler processing.

Half-bag idea: When the data is not enough, first save it until the decoding conditions are satisfied, put into the array. Sent to the business handler processing.

And the realization of this logic is our protagonist today: Bytetomessagedecoder.

The name means a decoder that converts bytes into messages. The name of a person. And he is also an inbound handler, so we still start with his Channelread method.

1. Channelread method

The streamlined code is as follows:

 Public void Channelread(Channelhandlercontext ctx, Object msg)throwsException {//Fetch a list from the object poolCodecoutputlist out = codecoutputlist.newinstance();    Bytebuf data = (BYTEBUF) msg; First = cumulation = =NULL;if(first) {//First time decodingcumulation = data;//Cumulative}Else{///second decoding, append data to cumulation and release datacumulation = Cumulator.cumulate(CTX.Alloc(), cumulation, data); }after receiving the appended cumulation, call the Decode method to decode    //During decoding, the Firechannelread method is called, the main purpose is to decode the contents of the accumulated area into the array    Calldecode(CTX, cumulation, out);//If the accumulation area has no readable bytes    if(cumulation! =NULL&&!cumulation.isreadable()) {//The number of times zeroedNumreads =0;//Release accumulation areaCumulation.Release();//wait for GCcumulation =NULL; }//If more than 16 times, the compression accumulation area, mainly will have read the data discarded, will readindex zero.     Else if(+ + numreads >= discardafterreads) {numreads =0;discardsomereadbytes(); }intSize = out.size();//If no data has been inserted into the arrayDecodewasnull =!out.insertsincerecycled();///loop array, send data to subsequent handler, if the array is empty, it will not call    Firechannelread(CTX, out, size);//Clears the contents of the array and restores the array's subscript to the originalOut.Recycle();}

The landlord has written a note in the method, but still talk about the main steps:

1. Remove an empty array from the object pool.
2. Determine if the member variable is used for the first time (note that since the member variable is used, the handler cannot be handler. Writes the data passed in the unsafe into this cumulation accumulation zone.
3. After writing to the accumulation area, call the Decode method of the subclass, attempt to decode the contents of the cumulative zone, and each successful decoding one, call the Channelread method of the latter node. If there is no decoding success, do nothing.
4. If the accumulation area has no unread data, the accumulation area is released.
5. If there are unread data, and the decoding is more than 16 times (the default), the accumulation area is compressed. Empties the read data, which is the Readindex set to 0.
6. Sets the value of Decodewasnull, which is ture if no data was inserted last time. This value, when calling the Channelreadcomplete method, triggers the Read method (not automatically read), attempts to read the data from the JDK's channel, and re-starts the previous logic. The main thing is to be afraid that if any data is not inserted, execution channelreadcomplete will miss the data.
7. Call the Firechannelread method to send the elements in the array to the following handler.
8. Empties the array. And back to the object pool.

Here are the detailed steps.

2. Remove an empty array from the object pool

Code:

@1Codecoutputlist out = codecoutputlist.newinstance();@2StaticCodecoutputlistnewinstance() {returnCodec_output_lists_pool.Get().getorcreate();}@3Private Static FinalFastthreadlocal<codecoutputlists> Codec_output_lists_pool =NewFastthreadlocal<codecoutputlists> () {@Override            protectedCodecoutputlistsInitialValue()throwsException {//Codecoutputlist per Thread is cached.                return New codecoutputlists( -); }        };@4codecoutputlists(intnumelements) {elements =NewCodecoutputlist[mathutil.Safefindnextpositivepoweroftwo(numelements)]; for(inti =0; I < elements.length; ++i) {//Size of should is good enough for the majority of all users as a initial capacity.Elements[i] =New codecoutputlist( This, -); } count = elements.length; Currentidx = elements.length; Mask = elements.length-1;}@5Private codecoutputlist(Codecoutputlistrecycler recycler,intSize) { This.recycler= Recycler; Array =NewObject[size];}@6 PublicCodecoutputlistgetorcreate() {if(Count = =0) {//Return a new codecoutputlist which won't be cached. We use a size of 4 to keep the overhead        //Low.        return New codecoutputlist(Noop_recycler,4); }--count;intIDX = (CURRENTIDX-1) & mask;    Codecoutputlist list = Elements[idx]; Currentidx = idx;returnList;}

The code is divided into 1,2,3,4,5, 6 steps.

1. static method invocation.
2. Remove a Codecoutputlists object from the fastthreadlocal and remove a list from the collection. That is, the list has a list. Can be understood as a double array.
3. Call Fastthreadlocal's InitialValue method to return a Codecoutputlists object.
4. Creates an array. The array size defaults to 16, looping through the codecoutputlist elements. Set the Count,currentidx, Mask property.
5. Create a Codecoutputlist object, this recycler is his parent codecoutputlists, and creates a default 16 empty array.
6. The first entry to count is not 0, it should be 16, and then the count-1, and the subscript with the operation out of the Lists, gets the contents of the subscript. A List. All parameters are restored when the Recycle method is called back to the object pool.

Since this getorcreate method is used by multiple places in a thread, 16 is a statistical value. When 16 is not enough, a new List is created. That is, the logic of Count = = 0. and the operation of & Mask is a modulo operation.

3. Write to the cumulative zone

The code is as follows:

cumulation = cumulator.cumulate(ctx.alloc(), cumulation, data);

This cumulator default is a cumulator type of merge_cumulator, the most important of which is to override the Cumulate method:

 Public Static FinalCumulator Merge_cumulator =New Cumulator() {@Override     PublicBytebufcumulate(Bytebufallocator alloc, bytebuf cumulation, bytebuf in) {FinalBYTEBUF buffer;if(cumulation.Writerindex() > Cumulation.maxcapacity()-In.readablebytes()                || Cumulation.refcnt() >1|| Cumulation.isreadonly()) {buffer =expandcumulation(Alloc, cumulation, in.)readablebytes()); }Else{buffer = cumulation; } buffer.writebytes(in); In.Release();returnBuffer }};

You can see this method, mainly to write the contents of the Bytebuf passed by Unsafe.read to the cumulation accumulation area, and then release the old content, because this variable is a member variable, so you can call the Channelread method write multiple times.

At the same time this method also takes into account the expansion of the problem, in general is copy.

Of course, there is a cumulator instance in Bytetomessagedecoder, called composite_cumulator, mixed accumulation. Because the Cumulate method of the last instance uses a memory copy, the use of mixed memory is provided here. Performance is better than copying, but it's also more complex.

4. Role of the Decode method

After the data has been chased into the accumulation area, the Decode method needs to be called to decode the code as follows:

@1Calldecode(CTX, cumulation, out);@2 protected void Calldecode(Channelhandlercontext ctx, bytebuf in, list<object> out) {//If the cumulative zone has readable bytes     while(In.isreadable()) {intoutsize = out.size();//Last loop successfully decoded        if(Outsize >0) {//Call the Channelread method behind the business handler            Firechannelread(CTX, out, outsize);//Place size at 0Out.Clear();//            if(CTX.isremoved()) { Break; } outsize =0; }//To get the number of readable bytes        intOldinputlength = in.readablebytes();////Call the Decode method to place the successfully decoded data into an out-of-the-box array, possibly deleting the current node and sending the data to the last handler before deleting it        decoderemovalreentryprotection(CTX, in, out);//decode ()        if(CTX.isremoved()) { Break; }if(outsize = = out.)size()) {if(Oldinputlength = = in.readablebytes()) { Break; }Else{Continue; }        }if(Issingledecode()) { Break; }    }}

The main logic of the method: As long as the accumulation area has unread data, the loop is read.

1. Calling the Decoderemovalreentryprotection method, internally calling the Decode method of the subclass override, it is clear that this is a template pattern. The logic of the Decode method is to decode the contents of the accumulated area according to the Convention, and if it is successfully decoded, it is added to the array. At the same time, the method also checks the state of the handler, if the pipeline is removed, the contents of the accumulated area are flushed directly to the handler in the back.

2. If the Context node is removed, the loop is ended directly. If the array size before decoding is equal to the size of the decoded array, and the number of readable bytes in the cumulative zone does not change, it means that the read does nothing and ends directly. If the number of bytes changes, the array does not increase, but it does read the bytes and then continues reading.

3. If the above is judged, the array reads the data, but if the readindex of the accumulated area does not change, an exception is thrown, indicating that no data is being read, but the array is incremented, and the operation of the subclass is not correct.

4. If it is a single decoder, the decoding is done directly.

So the key to this code is that the subclass needs to rewrite the Decode method to decode the accumulated data correctly and add it to the array. Each time a success is added, the Firechannelread method is called and the data in the array is passed to the following handler. When you are finished, set the size of the array to 0.

So, if your business handler in this place may be called multiple times. It may also not be called at a time. Depends on the values in the array. Of course, if the decoding handler is removed, all data from the accumulated area will be brushed to the back handler.

5. The rest of the logic

The logic above is the main logic of the decoder:

Reads the data of the Read method into the accumulation area, decodes the accumulated data using the decoder, decodes the successful one into an array, and passes the data from the array to the subsequent handler.

From the above logic, unless handler is removed, the subsequent handler method is not invoked, that is, the decoding rules of the decoder are not passed to the subsequent handler.

Then look at the logic behind it, mainly in the finally block:

1. If the accumulation area has no readable data, the counter is zeroed and the accumulation area is released.
2. If the above conditions are not met, and the counter exceeds 16 times, the content of the accumulation area is compressed, and the compressed method is to delete the read data. Place the Readindex at 0. Do you remember BYTEBUF's pointer structure?

This will save some memory, but this can cause some memory duplication process, with performance loss as the premise.

1. Record the Decodewasnull attribute, the value of which is determined by whether you have successfully inserted data into the array, if inserted, it is Fasle, no insert, he is true. The effect of this value is that when the Channelread method is finished, the Channelreadcomplete method that executes the decoder (if you have not rewritten it) will determine the value:

If true, the Autoread property will be judged, and if it is false, then Netty think there is still data to read, otherwise the array is always empty? Read from the Socket on the active call to the Read method.

1. Call the Firechannelread method to attempt to send the data in the array to a later handler. Why do you do this? As a matter of principle, when this step, the array can not be empty, why is it so cautious to send again?

A: If it is a single decoder, it needs to be sent, so the word decoder is no longer sent in the Calldecode method.

1. Finally, the array is returned to the object pool. and empties the array contents.

The last line of Recycler.recycle (this), there are two results, if the codecoutputlists recycle method, the contents are as follows:

Restores the array subscript to count + +, which indicates that an object is available.

There is a second, when 16 arrays are not enough, it is necessary to create a new one, embodied in the Getorcreate method. The recycler in the constructor is an empty object. Let's look at this object:

When the Recycle method is called, nothing is done. Wait for GC to recycle. Because this is not a reference to a pool of objects.

Well, here, the main function of the Bytetomessagedecoder decoder is read out.

5. Summary

It can be said that Bytetomessagedecoder is the core of the decoder, Netty used the template mode here, the way to subclass extension is Decode method.

The main logic is to put all the data into the cumulative area, the subclass from the accumulation of data extracted from the accumulated area into an array, Bytetomessagedecoder will loop array calls the following handler method, the data frame sent to the business handler. Complete the decoding logic for this.

In this way, whether it is a sticky bag or unpacking, it can be perfectly implemented.

There are some small details:

1. For example, the decoder can be single-time.
2. If the decoding has not been successful, then the data will not reach the back of the handler. Unless the decoder is removed from pipeline.
3. Like other Netty modules, the concept of object pooling is used here, the array is stored in the thread-safe ThreadLocal, the default is 16, and when not enough, a new one is created and then recycled by GC.
4. When the array has never successfully added data, and the program does not open Autoread, it actively calls the Read method. Attempt to read data.

Netty all decoders can be extended on this class, depending on the implementation of the decode. Just follow the Bytetomessagedecoder convention.

Good luck!!!!

Netty Decoder Abstract Parent class Bytetomessagedecoder source parsing