Solving the problem of sticky packets in TCP transmission in Golang

solving the problem of sticky packets in TCP transmission in Golang

Author: Yu Dongwei
Email:usher.yue@gmail.com What is a sticky bag.

Recently in writing https://github.com/UsherYue/ActivedRouter (a HTTP/HTTPS reverse proxy service) encountered a sticky packet problem,

If there is a network programming of the small partners should be aware of the sticky packet problem, for example: the client in the kimono

Use JSON-formatted packets for communication. Then the data interaction process for the client and server should look like this:

Client Send JSON data-> via network->server reveive data->server Decode json->done (only one Json packet in one interaction)

The above process we assume the number of intermediate interactions sent from the client to the server to receive this one-time sexual action

It is a completed JSON packet, so our program will work properly.

But the reality is not what we think, because of the characteristics of the TCP protocol, and the complexity of the network environment

Variable, and the server to the client's data receive processing is not timely, etc., will cause network transmission

Sticky packs appear during the process. That is, when the server does a data read, we assume that this

The packet is a complete JSON packet, but in fact he does, 2 JSON packets, 3

JSON packets, 2.5 JSON packets, that's what we call sticky packs.
If you don't understand it then look at the picture below.

How do we solve sticky bag problems?

In the development process, we usually define a fixed-length buffer to store the packets from the client connection when the server side receives the data, and then deserialize the packet, so to solve this problem we need to do some hands and feet from the time of sending and receiving the data, thinking as follows:

The Client Send Json data-> calls the encapsulation method to encapsulate the data into a fixed format packet, through thenetwork->server reveive data-> Call the unpack method to remove all Json packets from the packet, and stitch the remaining truncated data to the next incoming packet->server Decode JSON->done (only one JSON packet in one interaction)

I implemented a packet encapsulation code in Golang, which can be used directly as follows:

Package Packet Import ("bytes" "Encoding/binary") const (Defaule_header = "[**********]" DE Fault_header_length = Default_save_data_length = 4) type Packet struct {HEADER string Headerl Engh int32 savedatalength int32 Data []byte}//set delimiter header func (self *packet) setheader (h Eader string) *packet {self. Header = Header self. Headerlengh = Int32 (Len ([]byte header)) return self}//create default package func newdefaultpacket (data []byte) *p Acket {return &packet{defaule_header, default_header_length, Default_save_data_length, DATA}}//convert to net Package func (self *packet) Packet () []byte {return append (append ([]byte] (self. Header), self. Inttobytes (Int32 (Len (self). Data)), self.
Data ...) }//return value is sticky data func (self *packet) unpacket (Readerchannel Chan []byte) []byte {datalen: = Int32 (len (self.) Data)) var i int32 for i = 0; i < Datalen;i++ {//termiate for loops when the remaining data is insufficient. If Datalen < i+self. Headerlengh+self. savedatalength {break}//find Header if string (self. Data[i:i+self. Headerlengh]) = = self. Header {savedatalenbeginindex: = i + self. Headerlengh Actualdatalen: = self. Bytestoint (self. Data[savedatalenbeginindex:savedatalenbeginindex+self. Savedatalength])//the remaining data is less than one package if Datalen < i+self. Headerlengh+self. Savedatalength+actualdatalen {break}//get a packet packagedata: = S Elf. Data[savedatalenbeginindex+self. Savedatalength:savedatalenbeginindex+self. Savedatalength+actualdatalen]//send pacakge data to reader channel Readerchannel <-Packagedat A//get next package index i + = self. Headerlengh + self.
 Savedatalength + actualDataLen-1}   }//reach the End If I >= datalen {return []byte{}}//returns the remaining data retur N Self. Data[i:]} func (self *packet) inttobytes (i int32) []byte {bytebuffer: = bytes. Newbuffer ([]byte{}) binary. Write (Bytebuffer, Binary. Bigendian, I) return Bytebuffer.bytes ()} func (self *packet) bytestoint (data []byte) int32 {var val int32 b Ytebuffer: = bytes. Newbuffer (data) binary. Read (Bytebuffer, Binary.
 Bigendian, &val) return val}

The client implements the pseudo code code as follows:

  DataPackage: = Newdefaultpacket ([]byte (jsonstring)). Packet ()
  client.write (dataPackage)

The server implements the pseudo-code code as follows:

  Declare a pipe for receiving unpacked data
    Readerchannel: = Make (chan []byte, 1024x768)
    //store truncated data
    R Emainbuffer: = Make ([]byte, 0)
    //read unpackage data from buffered channel
    go func (reader Chan []byte] {
        for {
            Packagedata: = <-reader
            //....balabala ...
        }
    } (Readerchannel)
  Remainbuffer =   newdefaultpacket (append (Remainbuffer,recvdata)). Unpacket (Readerchannel)