This is a creation in Article, where the information may have evolved or changed.
What is the problem with TCP sticky packets and why TCP sticky packets are generated is not discussed in this article. This article uses Golang's Bufio. Scanner to implement custom protocol unpacking.
Protocol Packet Definition
This article simulates a log server that receives a packet that the client has uploaded and displays it
type Package struct {
Version [2]byte // 协议版本,暂定V1
Length int16 // 数据部分长度
Timestamp int64 // 时间戳
HostnameLength int16 // 主机名长度
Hostname []byte // 主机名
TagLength int16 // 标签长度
Tag []byte // 标签
Msg []byte // 日志数据
}
There is nothing good to say in the Protocol definition section, according to the specific business logic definition.
Data packaging
Because the TCP protocol is a language-independent protocol, it is not possible to send the protocol packet structure directly to the TCP connection, only send the byte stream data, so it is necessary to implement the data encoding. Fortunately, Golang provides binary to help us implement network byte encoding.
func (p *Package) Pack(writer io.Writer) error {
var err error
err = binary.Write(writer, binary.BigEndian, &p.Version)
err = binary.Write(writer, binary.BigEndian, &p.Length)
err = binary.Write(writer, binary.BigEndian, &p.Timestamp)
err = binary.Write(writer, binary.BigEndian, &p.HostnameLength)
err = binary.Write(writer, binary.BigEndian, &p.Hostname)
err = binary.Write(writer, binary.BigEndian, &p.TagLength)
err = binary.Write(writer, binary.BigEndian, &p.Tag)
err = binary.Write(writer, binary.BigEndian, &p.Msg)
return err
}
The output target of the Pack method is IO. Writer, facilitates the interface extension, as long as the implementation of the interface can encode data to write. Binary. Bigendian is a byte-order, this article is not discussed for the time being, readers who need it can find their own data research.
Data unpacking
The unpacking needs to parse the TCP packet into the struct, and then why you need to add several data-independent length fields.
Func(P *package)Unpack(Reader IO. Reader)Error {
var err error
Err = binary. Read (reader, Binary.) Bigendian, &p.version)
Err = binary. Read (reader, Binary.) Bigendian, &p.length)
Err = binary. Read (reader, Binary.) Bigendian, &p.timestamp)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostnamelength)
P.hostname = Make ([]byte, p.hostnamelength)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostname)
Err = binary. Read (reader, Binary.) Bigendian, &p.taglength)
P.tag = Make ([]byte, p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.tag)
P.msg = Make ([]byte, p.length-8-2-p.hostnamelength-2-p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.msg)
return Err
}
Because the host name, label this data is not fixed length, so it takes two bytes to identify the length of the data, otherwise read only to know that a total data length is unable to distinguish between host name, tag name, log data.
Problem solving for sticky packets of packets
The above only solves the encoding/decoding problem, provided that the received packets do not produce a sticky packet problem, to solve the sticky packet is to correctly split the data in the byte stream. There are generally the following practices:
Fixed-length separation (maximum per packet size) The disadvantage is that when data is insufficient, transport resources are wasted
The disadvantage of separating specific characters (e.g. \ r \ n) is that if there is \ r \ n In the body, it can cause problems
Add a length field to a packet (this article uses)
Golang provides the Bufio. Scanner to solve the problem of sticky packets.
Scanner: = Bufio. Newscanner (reader)Reader for the IO implementation. Reader interface objects, such as net. Conn
Scanner. Split (Func(Data []BYTE, ateofbool(Advanceint, token []BYTE, err error) {
If!ateof && data[0] = =' V ' {Since the header of the packet we define is the first two-byte version number, only packets that begin with V are processed
IfLen (data) >4 {//If the data received is >4 bytes (2-byte version number + 2-byte packet length)
length: = int16 ( 0)
binary. Read (bytes. Newreader (data[ 2: 4]), binary. Bigendian, &length) //Read packet 3–4 bytes (int16) = Data part length
if int (length) + 4 <= len (data) { //If the data body length read to + 2-byte version number + 2-byte data length does not exceed the data read ( In fact, it is a complete resolution of a package)
return int (length) + 4, data[: int (length) + 4], Nil
}
}
return
})
//Print received packets
For scanner. Scan () {
scannedpack: = new (Package)
scannedpack.unpack (bytes. Newreader (scanner. Bytes ()))
log. Println (scannedpack)
}
The core of this paper lies in scanner. Split method, which is used to parse the TCP packet
Full source
Package Main
Import (
"Bufio"
"Bytes"
"Encoding/binary"
"FMT"
"IO"
"Log"
"OS"
"Time"
)
Type Packagestruct {
Version [2]ByteProtocol version
LengthInt16Data part length
TimestampInt64Time stamp
HostnamelengthInt16Host name length
Hostname []ByteHost Name
TaglengthInt16Tag length
Tag []ByteTag
MSG []ByteData part length
}
Func(P *package)Pack(writer Io.) Writer)Error {
var err error
Err = binary. Write (writer, binary.) Bigendian, &p.version)
Err = binary. Write (writer, binary.) Bigendian, &p.length)
Err = binary. Write (writer, binary.) Bigendian, &p.timestamp)
Err = binary. Write (writer, binary.) Bigendian, &p.hostnamelength)
Err = binary. Write (writer, binary.) Bigendian, &p.hostname)
Err = binary. Write (writer, binary.) Bigendian, &p.taglength)
Err = binary. Write (writer, binary.) Bigendian, &p.tag)
Err = binary. Write (writer, binary.) Bigendian, &p.msg)
return err
}
Func(P *package)Unpack(Reader IO. Reader)Error {
var err error
Err = binary. Read (reader, Binary.) Bigendian, &p.version)
Err = binary. Read (reader, Binary.) Bigendian, &p.length)
Err = binary. Read (reader, Binary.) Bigendian, &p.timestamp)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostnamelength)
P.hostname =Make ([]BYTE, p.hostnamelength)
Err = binary. Read (reader, Binary.) Bigendian, &p.hostname)
Err = binary. Read (reader, Binary.) Bigendian, &p.taglength)
P.tag =Make ([]BYTE, p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.tag)
P.msg =Make ([]BYTE, p.length-8-2-p.hostnamelength-2-p.taglength)
Err = binary. Read (reader, Binary.) Bigendian, &p.msg)
return err
}
Func(P *package)String()string {
Return to FMT. Sprintf ("Version:%s length:%d timestamp:%d hostname:%s tag:%s msg:%s",
P.version,
P.length,
P.timestamp,
P.hostname,
P.tag,
P.msg,
)
}
FuncMain() {
Hostname, ERR: = OS. Hostname ()
If Err! =Nil {
Log. Fatal (ERR)
}
Pack: = &package{
Version: [2]byte{' V ',' 1 '},
Timestamp:time. Now (). Unix (),
Hostnamelength:Int16 (Len (hostname)),
Hostname: []BYTE (hostname),
Taglength:4,
TAG: []Byte"Demo"),
MSG: []BYTE ("The time is now:" + times. Now (). Format ("2006-01-02 15:04:05")),
}
Pack. Length =8 +2 + pack. Hostnamelength +2 + pack. Taglength +Int16 (Len (Pack. MSG))
BUF: =New (bytes. Buffer)
Write four times to simulate the TCP sticky effect
Pack. Pack (BUF)
Pack. Pack (BUF)
Pack. Pack (BUF)
Pack. Pack (BUF)
Scanner
Scanner: = Bufio. Newscanner (BUF)
Scanner. Split (Func(Data []BYTE, ateofbool(Advanceint, token []BYTE, err error) {
If!ateof && data[0] = =' V ' {
IfLen (data) >4 {
Length: = Int16 ( 0)
binary. Read (bytes. Newreader (data[ 2: 4]), binary. Bigendian, &length)
if int (length) + 4 <= len (data) {
return int (length) + 4, data[: int (length) + 4], Nil
}
}
return
})
For scanner. Scan () {
scannedpack: = new (package)
scannedpack.unpack (bytes. Newreader (scanner. Bytes ()))
log. Println (scannedpack)
}
if err: = Scanner. ERR (); Err!= nil {
log. Fatal ( "Invalid Packet")
}
}
Written in the last
Golang as a powerful network programming language, implementing a custom protocol is very important, and actually implementing a custom protocol is not difficult, the following several steps:
Packet encoding
Packet decoding
Handling TCP Sticky Packet issues
Wire Break re-connect (can use heartbeat) (not required)
Read the original