Solving the problem of sticky packets in TCP transmission in Golang

This is a creation in Article, where the information may have evolved or changed.

Solving the problem of sticky packets in TCP transmission in Golang

Author: Yu Dongwei
Email:usher.yue@gmail.com

What is a sticky bag?

Recently in writing Https://github.com/UsherYue/ActivedRouter (a HTTP/HTTPS reverse proxy service) encountered a sticky packet problem,

If there is a network programming of the small partners should be aware of the sticky packet problem, for example: the client in the kimono

Use JSON-formatted packets for communication. Then the data interaction process for the client and server should look like this:

Client Send JSON data-> via network->server reveive data->server Decode json->done (only one Json packet in one interaction)

The above process we assume the number of intermediate interactions sent from the client to the server to receive this one-time sexual action

It is a completed JSON packet, so our program will work properly.

But the reality is not what we think, because of the characteristics of the TCP protocol, and the complexity of the network environment

Variable, and the server to the client's data receive processing is not timely, etc., will cause network transmission

Sticky packs appear during the process. That is, when the server does a data read, we assume that this

The packet is a complete JSON packet, but in fact he does, 2 JSON packets, 3

JSON packets, 2.5 JSON packets, that's what we call sticky packs.
If you don't understand that, then look.

How do we solve sticky bag problems?

In the development process, we usually define a fixed-length buffer to store the packets from the client connection when the server side receives the data, and then deserialize the packet, so to solve this problem we need to do some hands and feet from the time of sending and receiving the data, thinking as follows:

The Client Send Json data-> calls the encapsulation method to encapsulate the data into a fixed format packet, through thenetwork->server reveive data-> Call the unpack method to remove all Json packets from the packet, and stitch the remaining truncated data to the next incoming packet->server Decode JSON->done (only one JSON packet in one interaction)

I implemented a packet encapsulation code in Golang, which can be used directly as follows:

 PackagePacketImport("bytes"    "Encoding/binary")Const(Defaule_header ="[**********]"Default_header_length = ADefault_save_data_length =4)typePacketstruct{HeaderstringHeaderlenghInt32SavedatalengthInt32Data []byte}//set delimiter Headerfunc(Self *packet) SetHeader (headerstring) *packet {self. Header = Header self. Headerlengh =Int32(Len([]byte(header)))returnSelf//create default PackagefuncNewdefaultpacket (data []byte) *packet {return&packet{defaule_header, Default_header_length, Default_save_data_length, DATA}}//convert to net packagefunc(Self *packet) Packet () []byte{return Append(Append([]byte(self.) Header), self. Inttobytes (Int32(Len(self.) Data)), self. Data ...)}//return value is sticky datafunc(Self *packet) Unpacket (ReaderchannelChan[]byte) []byte{Datalen: =Int32(Len(self.) Data))varIInt32     fori =0; i < Datalen; i++ {//termiate for Loop while the remaining data is insufficient.        ifDatalen < I+self. Headerlengh+self. Savedatalength { Break}//find Header        if string(self.) Data[i:i+self. Headerlengh]) = = self. Header {savedatalenbeginindex: = i + self. Headerlengh Actualdatalen: = self. Bytestoint (self. Data[savedatalenbeginindex:savedatalenbeginindex+self. Savedatalength])//the remaining data is less than one package            ifDatalen < I+self. Headerlengh+self. Savedatalength+actualdatalen { Break}//get a packetPackagedata: = self. Data[savedatalenbeginindex+self. Savedatalength:savedatalenbeginindex+self. Savedatalength+actualdatalen]//send pacakge data to reader channelReaderchannel <-Packagedata//get Next Package indexi + = self. Headerlengh + self. Savedatalength + Actualdatalen-1}    }//reach the end    ifI >= datalen {return[]byte{}    }//returns the remaining data    returnSelf. Data[i:]}func(Self *packet) Inttobytes (iInt32) []byte{bytebuffer: = bytes. Newbuffer ([]byte{}) binary. Write (Bytebuffer, Binary. Bigendian, I)returnBytebuffer.bytes ()}func(Self *packet) Bytestoint (data []byte)Int32{varValInt32Bytebuffer: = bytes. Newbuffer (data) binary. Read (Bytebuffer, Binary. Bigendian, &val)returnVal

The client implements the pseudo code code as follows:

  dataPackage := NewDefaultPacket([]byte(jsonString)).Packet()  Client.Write(dataPackage)

The server implements the pseudo-code code as follows:

  //declare a pipe for receiving unpacked dataReaderchannel: = Make(Chan[]byte,1024x768)//store truncated dataRemainbuffer: = Make([]byte,0)//read unpackage data from buffered channel    Go func(ReaderChan[]byte) { for{packagedata: = <-reader//....balabala ....}} (Readerchannel) Remainbuffer = Newdefaultpacket (Append(Remainbuffer,recvdata)). Unpacket (Readerchannel)