How to use IO package in Go

Objective

In Go, input and output operations are implemented using primitives that simulate data into a readable or writable stream of bytes.
To do this, the Go io package provides io.Reader and io.Writer interfaces for data input and output, respectively,

Go official provides some APIs to support the operation of memory structure , file , network connection and other resources.
This article focuses on how to implement a standard library io.Reader and io.Writer two interfaces to complete streaming data.

io.Reader

io.ReaderRepresents a reader that reads data from a resource into a transport buffer. In a buffer, data can be streamed and used.

For a type to be used as a reader, it must implement io.Reader the only method of the interface Read(p []byte) .
In other words, as long as Read(p []byte) it is implemented, it is a reader.

type Reader interface {    Read(p []byte) (n int, err error)}

Read()The method has two return values, one is the number of bytes read, and the other is the error when an error occurs.
Also, if the contents of the resource have all been read, an error should be returned io.EOF .

Using Reader

Readerstreaming data can be easily used. The inside of the Reader method is a loop called, each iteration, which reads a piece of data from the data source into the buffer p (that is, the read parameter p) until the io.EOF error is returned to stop.

Here is a simple example, by string.NewReader(string) creating a string reader and then streaming to read by byte:

func main() {    reader := strings.NewReader("Clear is better than clever")    p := make([]byte, 4)    for {        n, err := reader.Read(p)        if err != nil{            if err == io.EOF {                fmt.Println("EOF:", n)                break            }            fmt.Println(err)            os.Exit(1)        }        fmt.Println(n, string(p[:n]))    }}

输出打印的内容：4 Clea4 r is4  bet4 ter 4 than4  

As you can see, the last N value returned may be less than the buffer size.

Implement a Reader yourself

The previous section was implemented using a reader from the standard library io.Reader .
Now, let's see how to implement one yourself. Its function is to filter out non-alphabetic characters from the stream.

Type Alphareader struct {//resource SRC string//current read to position cur int}//Create an instance func Newalphareader (src string) *alph  Areader {return &alphareader{src:src}}//filter function func alpha (r byte) byte {if (R >= ' A ' && r <= ' Z ') || (R >= ' a ' && r <= ' z ') {return R} return 0}//Read method func (a *alphareader) read (p []byte) (int, error) {//Current position >= string length description has been read to the end to return EOF if A.cur >= len (a.src) {return 0, Io. EOF}//X is the remaining unread length x: = Len (a.src)-a.cur n, bound: = 0, 0 if x >= len (p) {//The remaining length exceeds the buffer size, stating this    The second can completely fill the buffer bound = Len (p)} else if x < Len (p) {//The remaining length is less than the buffer size, use the remaining length output, the buffer does not fill bound = x} BUF: = Make ([]byte, bound) for n < bound {//read one byte at a time, execute the filter function if char: = Alpha (a.src[a.cur]); Char! = 0 {Buf[n] = char} n++ a.cur++}//Copy processed BUF content to P, copy (P, buf) Retu RN N, nil}func Main () {reader: = NEWalphareader ("hello!    It ' s 9am, where is the sun? ") P: = Make ([]byte, 4) for {n, err: = Reader. Read (p) If err = = Io. EOF {break} fmt. Print (String (p[:n))} FMT. Println ()}

输出打印的内容：HelloItsamwhereisthesun

Combine multiple Readerto reuse and mask the complexity of the underlying implementation

The standard library has implemented many Reader.
Using one Reader as another Reader implementation is a common usage.
Doing so allows one to Reader reuse Reader The logic of another, shown below by updating alphaReader to accept io.Reader as its source.

type alphareader struct {//Alphareader combines the IO of the standard library. Reader Reader IO. Reader}func newalphareader (Reader io. Reader) *alphareader {return &alphareader{reader:reader}}func alpha (R byte) byte {if (R >= ' A ' && R <= ' Z ') | | (R >= ' a ' && r <= ' z ') {return R} return 0}func (a *alphareader) Read (P []byte) (int, error) {///This line of code calls IO. Reader N, err: = A.reader.read (P) if err! = Nil {return n, err} BUF: = make ([]byte, N) for I: = 0; I < n; i++ {if char: = Alpha (p[i]); Char! = 0 {Buf[i] = char}} copy (P, buf) return N, nil}f UNC main () {//Use implements standard library IO. The strings of the Reader interface. Reader as implementation reader: = Newalphareader (strings. Newreader ("hello! It's 9am, where is the sun?)) P: = Make ([]byte, 4) for {n, err: = Reader. Read (p) If err = = Io. EOF {break} fmt. Print (String (p[:n))} FMT. Println ()} 

Another advantage of doing this is the alphaReader ability to read from any reader implementation.
For example, the following code shows alphaReader how to os.File combine to filter out non-alphabetic characters in a file:

func main() {    // file 也实现了 io.Reader    file, err := os.Open("./alpha_reader3.go")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()        // 任何实现了 io.Reader 的类型都可以传入 newAlphaReader    // 至于具体如何读取文件，那是标准库已经实现了的，我们不用再做一遍，达到了重用的目的    reader := newAlphaReader(file)    p := make([]byte, 4)    for {        n, err := reader.Read(p)        if err == io.EOF {            break        }        fmt.Print(string(p[:n]))    }    fmt.Println()}

io.Writer

io.WriterRepresents a writer that reads data from a buffer and writes data to the target resource.

For a type to be used as a writer, the only method that must implement io.Writer the interfaceWrite(p []byte)
Similarly, as long as Write(p []byte) it is implemented, it is a writer.

type Writer interface {    Write(p []byte) (n int, err error)}

Write()The method has two return values, one is the number of bytes written to the target resource, and one is the error when the error occurs.

Using Writer

The standard library provides a number of types that have already been implemented io.Writer .
Here is a simple example that uses the bytes.Buffer type as io.Writer the write data to the memory buffer.

func main() {    proverbs := []string{        "Channels orchestrate mutexes serialize",        "Cgo is not Go",        "Errors are values",        "Don't panic",    }    var writer bytes.Buffer    for _, p := range proverbs {        n, err := writer.Write([]byte(p))        if err != nil {            fmt.Println(err)            os.Exit(1)        }        if n != len(p) {            fmt.Println("failed to write data")            os.Exit(1)        }    }    fmt.Println(writer.String())}

输出打印的内容：Channels orchestrate mutexes serializeCgo is not GoErrors are valuesDon't panic

Implement a Writer yourself

Let's implement a chanWriter custom named io.Writer , which writes its contents as a sequence of bytes channel .

type chanWriter struct {    // ch 实际上就是目标资源    ch chan byte}func newChanWriter() *chanWriter {    return &chanWriter{make(chan byte, 1024)}}func (w *chanWriter) Chan() <-chan byte {    return w.ch}func (w *chanWriter) Write(p []byte) (int, error) {    n := 0    // 遍历输入数据，按字节写入目标资源    for _, b := range p {        w.ch <- b        n++    }    return n, nil}func (w *chanWriter) Close() error {    close(w.ch)    return nil}func main() {    writer := newChanWriter()    go func() {        defer writer.Close()        writer.Write([]byte("Stream "))        writer.Write([]byte("me!"))    }()    for c := range writer.Chan() {        fmt.Printf("%c", c)    }    fmt.Println()}

To use this Writer, simply call it in the function main() writer.Write() (in a separate goroutine).
Because chanWriter the interface is also implemented io.Closer , the method is called writer.Close() to properly close the channel to avoid leaks and deadlocks.

ioOther useful types and methods in the package

As mentioned earlier, the Go standard library comes with a number of useful features and types that make it easy for my mom to use streaming IO.

os.File

Type os.File represents a file on the local system. It is implemented io.Reader and io.Writer , therefore, can be used in any IO context.
For example, the following example shows how to write a contiguous string slice directly to a file:

func main() {    proverbs := []string{        "Channels orchestrate mutexes serialize\n",        "Cgo is not Go\n",        "Errors are values\n",        "Don't panic\n",    }    file, err := os.Create("./proverbs.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()    for _, p := range proverbs {        // file 类型实现了 io.Writer        n, err := file.Write([]byte(p))        if err != nil {            fmt.Println(err)            os.Exit(1)        }        if n != len(p) {            fmt.Println("failed to write data")            os.Exit(1)        }    }    fmt.Println("file write done")}

It io.File can also be used as a reader to read the contents of a file from the local file system.
For example, the following example shows how to read a file and print its contents:

func main() {    file, err := os.Open("./proverbs.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()    p := make([]byte, 4)    for {        n, err := file.Read(p)        if err == io.EOF {            break        }        fmt.Print(string(p[:n]))    }}

标准输入、输入和错误

osThe package has three available variables os.Stdout , os.Stdin and os.Stderr , they are of the type *os.File , represented separately 系统标准输入 , 系统标准输出 and 系统标准错误 the file handle.
For example, the following code prints directly to standard output:

func main() {    proverbs := []string{        "Channels orchestrate mutexes serialize\n",        "Cgo is not Go\n",        "Errors are values\n",        "Don't panic\n",    }    for _, p := range proverbs {        // 因为 os.Stdout 也实现了 io.Writer        n, err := os.Stdout.Write([]byte(p))        if err != nil {            fmt.Println(err)            os.Exit(1)        }        if n != len(p) {            fmt.Println("failed to write data")            os.Exit(1)        }    }}

io.Copy()

io.Copy()It is easy to copy data from one Reader to another Writer.
It abstracts the for loop pattern (which we've already implemented above) and handles io.EOF and counts the bytes correctly.
Here are the simplified versions we implemented earlier:

func main() {    proverbs := new(bytes.Buffer)    proverbs.WriteString("Channels orchestrate mutexes serialize\n")    proverbs.WriteString("Cgo is not Go\n")    proverbs.WriteString("Errors are values\n")    proverbs.WriteString("Don't panic\n")    file, err := os.Create("./proverbs.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()    // io.Copy 完成了从 proverbs 读取数据并写入 file 的流程    if _, err := io.Copy(file, proverbs); err != nil {        fmt.Println(err)        os.Exit(1)    }    fmt.Println("file created")}

Then, we can also use io.Copy() function overrides to read from a file and print to the standard output of the previous program, as follows:

func main() {    file, err := os.Open("./proverbs.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()    if _, err := io.Copy(os.Stdout, file); err != nil {        fmt.Println(err)        os.Exit(1)    }}

io.WriteString()

This function allows us to easily write a string type to a writer:

func main() {    file, err := os.Create("./magic_msg.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()    if _, err := io.WriteString(file, "Go is fun!"); err != nil {        fmt.Println(err)        os.Exit(1)    }}

使用管道的 Writer 和 Reader

Type io.PipeWriter and io.PipeReader simulate IO operations in the memory pipeline.
The data is written to one end of the pipeline and is read at the other end of the pipe using a separate goroutine.
The following uses the io.Pipe() reader and writer that created the pipeline, and then copies the data from proverbs the buffer to io.Stdout :

func main() {    proverbs := new(bytes.Buffer)    proverbs.WriteString("Channels orchestrate mutexes serialize\n")    proverbs.WriteString("Cgo is not Go\n")    proverbs.WriteString("Errors are values\n")    proverbs.WriteString("Don't panic\n")    piper, pipew := io.Pipe()    // 将 proverbs 写入 pipew 这一端    go func() {        defer pipew.Close()        io.Copy(pipew, proverbs)    }()    // 从另一端 piper 中读取数据并拷贝到标准输出    io.Copy(os.Stdout, piper)    piper.Close()}

缓冲区 io

The package in the standard library bufio supports buffer IO operations and can easily handle text content.
For example, the following program reads the contents of a file row by line, separated by values '\n' :

func main() {    file, err := os.Open("./planets.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    defer file.Close()    reader := bufio.NewReader(file)    for {        line, err := reader.ReadString('\n')        if err != nil {            if err == io.EOF {                break            } else {                fmt.Println(err)                os.Exit(1)            }        }        fmt.Print(line)    }}

ioutil

ioA sub-package underneath the package utilio encapsulates some very handy features
For example, the following uses a function ReadFile to load the contents of a file into []byte .

package mainimport (  "io/ioutil"   ...)func main() {    bytes, err := ioutil.ReadFile("./planets.txt")    if err != nil {        fmt.Println(err)        os.Exit(1)    }    fmt.Printf("%s", bytes)}

Summarize

This article describes how to use io.Reader and io.Writer interface to implement streaming IO in a program.
After reading this article, you should be able to learn how to use io a package to implement a program that streams IO data.
Some of these examples show how to create your own type and implement io.Reader and io.Writer .

This is a simple introduction to the nature of the article, no extension open.
For example, we do not have deep file io, buffered io, network IO or formatted IO (saved for future writes).
I hope this article will give you an idea of what a common use of the go language is in streaming IO.

Thank you!