I was bufio by the idiot. Reader Little Pit

This is a creation in Article, where the information may have evolved or changed.

Recently, I'm using Go to do a small gateway service. PHP requests the Go TCP server, then go according to the command parameters to open multiple Goroutine to dispatch php-fpm execute different scripts and combine the results returned. Want to just take advantage of the convenience of goroutine concurrent execution logic, so simple and straightforward.

But in the test when PHP sent the socket JSON data has been explicitly truncated, and later found that I idiot the bufio.Reader pit, really speechless.

Problem recurrence

Because it is a long connection, PHP each time a piece of JSON, will add newline character \n segmentation. I took it for granted *bufio.Reader.ReadLine() . Just like the following code:

func handleConn(conn net.Conn) {    reader := bufio.NewReader(conn)    for {        // 读取一行数据，交给后台处理        line,_,err := reader.ReadLine()        if len(line) > 0{            fmt.Printf("ReadData|%d \n",len(line))            executeBytes(line)        }        if err != nil{            break        }    }    conn.Close()}

However, when the Phper test sent large JSON data, a clear truncation was found:

ReadData|4096|{"ename.com":........,"yunduo.com":{"check":[1// 又一次ReadData|4096|{"ename.com":........,"yunduo.com":{"getWhois"

You want JSON to be truncated before it's finished reading. I remember the default Bufio. The size of Reader is 4096, so bufio. Reader's behavior is to read a full buffer to return AH. I'll take a walk. I can't always write a big size.

    reader := bufio.NewReaderSize(conn,409600)

The test data sent by PHP is likely to be around 100k, averaging only <10k. My service side each time to open a 100k+ bufio.Reader too wasted. Finally become, open small cut, open big waste, I idiot.

How to Solve

Since bufio.Reader.ReadLine() I can't play anymore, I can only go back to the most primitive usage, such as the modified code:

func handleConn(conn net.Conn) {    buf := make([]byte, 4096)    var jsonBuf bytes.Buffer    for {        n, err := conn.Read(buf)        if n > 0 {            if buf[n-1] == 10 { // 10就是\n的ASCII                jsonBuf.Write(buf[:n-1]) // 去掉最后的换行符                executeBytes(jsonBuf.Bytes())                jsonBuf.Reset() // 重置后用于下一次解析            } else {                jsonBuf.Write(buf[:n])            }        }        if err != nil {            break        }    }    conn.Close()}

This is to read 4096 bytes each time, read \ n truncation, and the previous buffer into the data taken out together to the subsequent operations. This does not care about the data size, as long as the \ n delimiter is correct, there is no problem.

No business problems were detected and pprof did not see significant performance bottlenecks. That's it, I'm relieved.

It's idiot again.

bufio.Reader.ReadLine()Should not be so silly. If the buffer is full then return, how do I know if it's just full or read \ nthe return, in case the content is just buffer length! Go home from work to turn the source, I idiot again:

ReadLine is a low-level line-reading primitive. Most callers should use//readbytes (' \ n ') or ReadString (' \ n ') instead or use a scanner.////ReadLine tries to return a sin GLE line, not including the End-of-line bytes.//If the line was too long for the buffer then isprefix are set and the//be Ginning of the line is returned. The rest of the line is returned//from the future calls. Isprefix would be false if returning the last fragment//of the line. The returned buffer is a valid until the next call to//ReadLine. ReadLine either returns a Non-nil line or it returns an error,//never both.////... func (b *reader) ReadLine () (line [  ]byte, isprefix bool, err error) {line, err = B.readslice (' \ n ') if Err = = errbufferfull {//Handle the case        where "\ r \ n" straddles the buffer.            If len > 0 && line[len (line)-1] = = ' \ R ' {//Put the ' \ R ' back on BUF and drop it from line.    Let the next call to ReadLine check for "\ r \ n".        If B.R = = 0 {//should be unreachable panic ("bufio:tried to rewind past start of B    Uffer ")} b.r--line = Line[:len (line)-1]} return line, True, nil} ......}

I rub. True if the read out buffer is full, when it is not the \ n End isPrefix . I never noticed the meaning of this return value in the middle. I was really in the hole. In fact, the code can be written like this:

func handleConn(conn net.Conn) {    reader := bufio.NewReader(conn)    var jsonBuf bytes.Buffer    for {        // 读取一行数据，交给后台处理        line,isPrefix,err := reader.ReadLine()        if len(line) > 0{            jsonBuf.Write(line)            if !isPrefix{                executeBytes(jsonBuf.Bytes())                jsonBuf.Reset()            }        }        if err != nil{            break        }    }    conn.Close()}

RTFD

Conclusion:Read the fucking documentation