Golang solving the problem of TCP sticky packets

Source: Internet
Author: User
Tags unpack
This is a creation in Article, where the information may have evolved or changed.

What is the problem with TCP sticky packets and why TCP sticky packets are generated is not discussed in this article. This article uses Golang's Bufio. Scanner to implement custom protocol unpacking.

Protocol Packet Definition

This article simulates a log server that receives a packet that the client has uploaded and displays it

type Package struct {
 Version        [2]byte // 协议版本,暂定V1
 Length         int16   // 数据部分长度
 Timestamp      int64   // 时间戳
 HostnameLength int16   // 主机名长度
 Hostname       []byte  // 主机名
 TagLength      int16   // 标签长度
 Tag            []byte  // 标签
 Msg            []byte  // 日志数据
}

There is nothing good to say in the Protocol definition section, according to the specific business logic definition.

Data packaging

Because the TCP protocol is a language-independent protocol, it is not possible to send the protocol packet structure directly to the TCP connection, only send the byte stream data, so it is necessary to implement the data encoding. Fortunately, Golang provides binary to help us implement network byte encoding.

func (p *Package) Pack(writer io.Writer) error {
 var err error
 err = binary.Write(writer, binary.BigEndian, &p.Version)
 err = binary.Write(writer, binary.BigEndian, &p.Length)
 err = binary.Write(writer, binary.BigEndian, &p.Timestamp)
 err = binary.Write(writer, binary.BigEndian, &p.HostnameLength)
 err = binary.Write(writer, binary.BigEndian, &p.Hostname)
 err = binary.Write(writer, binary.BigEndian, &p.TagLength)
 err = binary.Write(writer, binary.BigEndian, &p.Tag)
 err = binary.Write(writer, binary.BigEndian, &p.Msg)
 return err
}

The output target of the Pack method is IO. Writer, facilitates the interface extension, as long as the implementation of the interface can encode data to write. Binary. Bigendian is a byte-order, this article is not discussed for the time being, readers who need it can find their own data research.

Data unpacking

The unpacking needs to parse the TCP packet into the struct, and then why you need to add several data-independent length fields.

Func(P *package)Unpack(Reader IO. Reader)Error {
var err error
Err = binary. Read (reader, Binary.) Bigendian, &p.version)
Err = binary. Read (reader, Binary.) Bigendian, &p.length)
Err = binary. Read (reader, Binary.) Bigendian, &p.timestamp)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostnamelength)
P.hostname = Make ([]byte, p.hostnamelength)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostname)
Err = binary. Read (reader, Binary.) Bigendian, &p.taglength)
P.tag = Make ([]byte, p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.tag)
P.msg = Make ([]byte, p.length-8-2-p.hostnamelength-2-p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.msg)
return Err
}

Because the host name, label this data is not fixed length, so it takes two bytes to identify the length of the data, otherwise read only to know that a total data length is unable to distinguish between host name, tag name, log data.

Problem solving for sticky packets of packets

The above only solves the encoding/decoding problem, provided that the received packets do not produce a sticky packet problem, to solve the sticky packet is to correctly split the data in the byte stream. There are generally the following practices:

    1. Fixed-length separation (maximum per packet size) The disadvantage is that when data is insufficient, transport resources are wasted

    2. The disadvantage of separating specific characters (e.g. \ r \ n) is that if there is \ r \ n In the body, it can cause problems

    3. Add a length field to a packet (this article uses)

Golang provides the Bufio. Scanner to solve the problem of sticky packets.

Scanner: = Bufio. Newscanner (reader)Reader for the IO implementation. Reader interface objects, such as net. Conn
Scanner. Split (Func(Data []BYTE, ateofbool(Advanceint, token []BYTE, err error) {
If!ateof && data[0] = =' V ' {Since the header of the packet we define is the first two-byte version number, only packets that begin with V are processed
IfLen (data) >4 {//If the data received is >4 bytes (2-byte version number + 2-byte packet length)
     length: =  int16 ( 0)
     binary. Read (bytes. Newreader (data[ 2: 4]), binary. Bigendian, &length)   //Read packet 3–4 bytes (int16) = Data part length
      if  int (length) + 4 <=  len (data) {  //If the data body length read to + 2-byte version number + 2-byte data length does not exceed the data read ( In fact, it is a complete resolution of a package)
        return  int (length) +  4, data[: int (length) + 4],  Nil
     
   }
 }
  return
})
//Print received packets
For scanner. Scan () {
 scannedpack: =  new (Package)
 scannedpack.unpack (bytes. Newreader (scanner. Bytes ()))
 log. Println (scannedpack)
}

The core of this paper lies in scanner. Split method, which is used to parse the TCP packet

Full source

Package Main
Import (
"Bufio"
"Bytes"
"Encoding/binary"
"FMT"
"IO"
"Log"
"OS"
"Time"
)
Type Packagestruct {
Version [2]ByteProtocol version
LengthInt16Data part length
TimestampInt64Time stamp
HostnamelengthInt16Host name length
Hostname []ByteHost Name
TaglengthInt16Tag length
Tag []ByteTag
MSG []ByteData part length
}
Func(P *package)Pack(writer Io.) Writer)Error {
var err error
Err = binary. Write (writer, binary.) Bigendian, &p.version)
Err = binary. Write (writer, binary.) Bigendian, &p.length)
Err = binary. Write (writer, binary.) Bigendian, &p.timestamp)
Err = binary. Write (writer, binary.) Bigendian, &p.hostnamelength)
Err = binary. Write (writer, binary.) Bigendian, &p.hostname)
Err = binary. Write (writer, binary.) Bigendian, &p.taglength)
Err = binary. Write (writer, binary.) Bigendian, &p.tag)
Err = binary. Write (writer, binary.) Bigendian, &p.msg)
return err
}
Func(P *package)Unpack(Reader IO. Reader)Error {
var err error
Err = binary. Read (reader, Binary.) Bigendian, &p.version)
Err = binary. Read (reader, Binary.) Bigendian, &p.length)
Err = binary. Read (reader, Binary.) Bigendian, &p.timestamp)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostnamelength)
P.hostname =Make ([]BYTE, p.hostnamelength)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostname)
Err = binary. Read (reader, Binary.) Bigendian, &p.taglength)
P.tag =Make ([]BYTE, p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.tag)
P.msg =Make ([]BYTE, p.length-8-2-p.hostnamelength-2-p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.msg)
return err
}
Func(P *package)String()string {
Return to FMT. Sprintf ("Version:%s length:%d timestamp:%d hostname:%s tag:%s msg:%s",
P.version,
P.length,
P.timestamp,
P.hostname,
P.tag,
P.msg,
)
}
FuncMain() {
Hostname, ERR: = OS. Hostname ()
If Err! =Nil {
Log. Fatal (ERR)
}
Pack: = &package{
Version: [2]byte{' V ',' 1 '},
Timestamp:time. Now (). Unix (),
Hostnamelength:Int16 (Len (hostname)),
Hostname: []BYTE (hostname),
Taglength:4,
TAG: []Byte"Demo"),
MSG: []BYTE ("The time is now:" + times. Now (). Format ("2006-01-02 15:04:05")),
}
Pack. Length =8 +2 + pack. Hostnamelength +2 + pack. Taglength +Int16 (Len (Pack. MSG))
BUF: =New (bytes. Buffer)
Write four times to simulate the TCP sticky effect
Pack. Pack (BUF)
Pack. Pack (BUF)
Pack. Pack (BUF)
Pack. Pack (BUF)
Scanner
Scanner: = Bufio. Newscanner (BUF)
Scanner. Split (Func(Data []BYTE, ateofbool(Advanceint, token []BYTE, err error) {
If!ateof && data[0] = =' V ' {
IfLen (data) >4 {
Length: = Int16 ( 0)
       binary. Read (bytes. Newreader (data[ 2: 4]), binary. Bigendian, &length)
        if  int (length) + 4 <=  len (data) {
          return  int (length) +  4, data[: int (length) + 4],  Nil
       
     }
    }
    return
 })
  For scanner. Scan () {
   scannedpack: =  new (package)
   scannedpack.unpack (bytes. Newreader (scanner. Bytes ()))
   log. Println (scannedpack)
 }
  if err: = Scanner. ERR (); Err!=  nil {
   log. Fatal ( "Invalid Packet")
 }
}

Written in the last

Golang as a powerful network programming language, implementing a custom protocol is very important, and actually implementing a custom protocol is not difficult, the following several steps:

    1. Packet encoding

    2. Packet decoding

    3. Handling TCP Sticky Packet issues

    4. Wire Break re-connect (can use heartbeat) (not required)

Read the original

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.