Some Golang tips

This is a creation in Article, where the information may have evolved or changed.

Today, I would like to introduce 3 Golang tips that I think are more enlightening, namely the following code snippets

• NSQ Select Write file and socket
• Sendfile in the IO module
• Handling of headers in Fasthttp

The select Read in NSQ

In the NSQ, you need to read the previous disk, or read directly from memory, the average person is to determine whether there is data in memory, however, nsq a way to use the SELECT statement, the CSP model used to the extreme.

Source File Links: channel.go

        select {        case msg = <-c.memoryMsgChan:  //尝试从内存中读取        case buf = <-c.backend.ReadChan(): //如果内存中没有，直接从磁盘上读取            msg, err = decodeMessage(buf)            if err != nil {                c.ctx.nsqd.logf("ERROR: failed to decode message - %s", err)                continue            }

Sendfile in the IO module

After a smart IO. Readfrom interface design, sendfile to the upper HTTP handler completely transparent, specifically called

    +----------------+    | http.    Servefile |    +--------+-------+             |     +--------+--------+      +----------------+            +---------------------------------+    | Os. File +------> IO.            Copy | | http.    Response | +--------+--------+      +--------+-------+            |             +-----------------------------+ |                        |                    | | | Net.             Tcpconn | |               | +--------v-------+ 2.   Has? | |             +-------------------------+ | |               |  | Io. CopyBuffer +---------> | | | Io. Readfrom | |               +-----+             | +--------+-------+            | | |     +---------------------+ | | |             |                        |                    | | | | |     Sednfile (syscall) | | | |             |                        |                    | | | |     +---------------------+ | | |             |                        |     |               | |     +-------------------------+ | |             |                        |                    | |     +-----------------------------+ |             |                        |                    |             +---------------------------------+     |   |   4. Do it! +--------v------+ 3.                                           yes!             |                             +---------------> Syscall <-----------------------------------------------------+ +----------------

1. The HTTP module is simply opened directly to the file, getting the file descriptor (filename descriptor)
2. The HTTP module calls IO. copy function, Io. The copy function begins to check whether reader writer is a special Readfrom,writeto interface

    func copyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error) {    // bla....    // Similarly, if the writer has a ReadFrom method, use it to do the copy.       if rt, ok := dst.(ReaderFrom); ok {      return rt.ReadFrom(src)      }

1. Complete the judgment and call net directly. Readfrom interface under the Tcpconn module, which is written in Sendfile

func (c *TCPConn) ReadFrom(r io.Reader) (int64, error) {        if n, err, handled := sendFile(c.fd, r); handled {        if err != nil && err != io.EOF {       err = &OpError{Op: "read", Net: c.fd.net, Source: c.fd.laddr, Addr: c.fd.raddr, Err: err}       }       return n, err       }    // skipped....

Such IO. Copy users actually do not know that they silently use the sendfile, while maintaining the interface consistency and very low coupling degree.

More in-depth can be a tour of Xie Da-interface

Fasthttp Handling of headers

One of the reasons why Fasthttp is faster than net.http is that Fasthttp does not fully parse all HTTP request header data. This practice is also known as lazyloading. First of all, let's start with the struct at the header.

type RequestHeader struct {        //bla.....    contentLength      int    contentLengthBytes []byte    method      []byte    requestURI  []byte    host        []byte    contentType []byte    userAgent   []byte    h     []argsKV    bufKV argsKV    cookies []argsKV    rawHeaders []byte}

Perhaps everyone is very strange, the request does not have a string, obviously method, host can use string Ah, this is because the string is immutable type, and read from the reader []byte is a mutable type, therefore, from the [] When byte is converted to string, it has copy and alloc behavior. While the amount of data is small, it has little impact, but when you build a high-concurrency system, the smaller data becomes much larger and more burdensome to the GC.

There is also a special argskv in the request.

type argsKV struct {    key   []byte    value []byte}

In fact, and the above reason is the same, Net.http used in the map[string]string to store a variety of other parameters, which requires alloc, in order to achieve zeroalloc,fasthttp used to traverse all the header parameters to return, In fact, there are certain considerations, that is, most of the header number is actually not many, and each time for short key comparison only need a number of CPU cycles, so it is reasonable tradeoff (see bytes. Equal assembly Implementation)

for []byte alloc optimization, refer to Dave Cheney's Five things that makes Gofast