Gracefully implement TCP compression transmission

This is a creation in Article, where the information may have evolved or changed.

The clustered, load-balanced RPC Framework RPCX supports a variety of serialization libraries, which can effectively reduce the size of the message body, but can further compress the byte slice of a string or picture. As fasthttp author Valyala described in his new open source project Httpteleport: 通过1G的带宽传输10G的数据 (exaggeration).

In order to reduce the transmission of data in the RPC transmission size, I do not affect the RPCX overall framework, based on reference to the implementation of Httpteleport, the net.TCPConn package, to achieve the compression/decompression function net.Conn , can effectively reduce bandwidth, saving the company's bandwidth costs, The following is a concrete implementation.

The first two compression formats are described.

zip is a commonly used compression format, and the Go Standard library provides its implementation. Zip Formerly known as deflate, the inventor of Fille Katz (Phil Katz), published the information in that format in January 1989. Zip usually uses the suffix name ". zip", which is in MIME format application/zip. Currently, the zip format belongs to one of several mainstream compression formats.

Snappy(formerly known as Zippy) is Google's LZ77-based approach to fast data compression and decompression programs written in the C + + language, and open source in 2011. Its target is not the maximum compression rate or compatibility with other compression libraries, but rather a very high speed and reasonable compression ratio. Use a single core of the Core™ i7 processor running in 64-bit mode, compression speed of up to MB/s, decompression speed of up to MB/s. Compression rate is 20-100% lower than gzip. Golang also provides the implementation of the snappy.

So in the tradeoff between compression ratio and speed, you can choose ZIP format compression or snappy format compression.

Define these types of formats:

12345678910

type byte Const (//Compressnone represents no compressioniota//compressflate represents zipcompressflate //Compresssnappy represents snappy Compresssnappy)

Then define the CompressConn type, which embeds an anonymous net.Conn type of field as net.Conn the wrapper, so it satisfies the net.Conn interface.
The r w Compresstype defines the compression type, which can be achieved by compressing read-write.

123456

type struct {Net. Connr io. Readerw io. Writercompresstype Compresstype}

net.Connread-write Methods for overwriting:

1234567

func (c *compressconn) Read (b []byteint, err error) {return C.r.read (b)}func (c *compressconn) Write (b []byteint, err error) {return c.w.write (b)}

NewCompressConnIs the secondary creation method:

12345678910111213141516171819202122232425262728

funcNEWCOMPRESSCONN (Conn net. Conn, Compresstype compresstype) net. Conn {cc: = &compressconn{conn:conn}r: = Io. Reader (CC. Conn)SwitchCompresstype { CaseCompressnone: CaseCompressflate:r = Flate. Newreader (R) CaseCompresssnappy:r = Snappy. Newreader (r)}CC.R = rw: = IO. Writer (CC. Conn)SwitchCompresstype { CaseCompressnone: CaseCOMPRESSFLATE:ZW, err: = Flate. Newwriter (W, Flate. Defaultcompression)ifErr! =Nil{Panic(FMT. Sprintf ("Bug:flate. Newwriter (%d) returned Non-nil err:%s ", Flate. Defaultcompression, err))}w = &WRITEFLUSHER{W:ZW} CaseCompresssnappy:w = Snappy. Newwriter (w)}cc.w = Wreturncc

For zip the format of the write, after writing we need to immediately Flush , so also need to wrap it:

1234567891011121314

type struct {W *flate. Writer}func (WF *writeflusher) Write (p []byte) (int, error) {n, err: = Wf.w.write (p)if Nil {return N, err}ifnil {return 0, err}return Nil}

This allows us to implement a read-write object that can be decompressed/compressed, and net.Conn you can NewCompressConn net.TCPConn wrap it in a normal way, and call it like a normal one net.Conn .

Update : I added support for ZSTD and LZ4.