Go mimics UNIX pipeline operations

This is a creation in Article, where the information may have evolved or changed.

This article extracts the Xu Xiwei "go, based on the language of connection and combination" part of the content, in order to facilitate understanding, I wrote a complete sample program on the back side to help understand, this article is to show the go in parallel programming of the great, but also to understand and learn closure of the living textbooks

--------------------------------------------------------------------------------------------------------------- ------------------------------------------------------

Let's talk about it from UNIX. The go language has deep roots in Unix and C language. The leaders of the go language have even dominated the design of Unix and C languages. Ken Thompson is even the originator of UNIX and C languages. The go language is also deeply influenced by the design philosophy of Unix and C language.

In the UNIX world, components are applications (apps), and each app can be roughly abstracted to:

• Input: stdin (standard input), params (command line arguments)
• Output: stdout (standard output)
• Protocol: text (data stream)

How are different apps (apps) connected? The answer is: pipe (pipeline). In the Unix world people are already familiar with this kind of thing:

App1 Params1 | APP2 PARAMS2

With pipelines (pipeline), you can convert an application's output (stdout) to another application's input (stdin). The more magical thing is that these applications are executed in parallel. App1 each output, it is immediately processed by the APP2. So the pipeline (pipeline) is the oldest, but also a very good parallel facility, simple and powerful.

It is important to note that different applications in the UNIX world are directly loosely coupled. The output from the upstream app is XML or JSON, and the downstream app needs to know, but there is no mandatory constraint. The same output, different downstream apps, may not even have the same understanding of the Protocol. For example, the upstream app outputs a piece of XML text that is a DOM tree for a downstream app, but only a multiline text for the LineCount program, an English article for the English word frequency Statistics program.

To facilitate understanding, we first try to simulate the entire UNIX pipeline (pipeline) mechanism in the go language. The first is the application (app), which we abstract as:

Func (in IO. Reader, out IO. Writer, args []string]

We press to correspond to the UNIX and Go Code relationship:

In other words, the Unix

App1 Params1 | APP2 PARAMS2

The corresponding go language is:

Pipe (Bind (App1, PARAMS1), bind (APP2, PARAMS2))

Where the BIND function is implemented as follows:

Func bind (    app func (in IO). Reader, out IO. Writer, args []string),    args []string] func (in IO. Reader, out IO. Writer) {    return func (in IO). Reader, out IO. Writer) {        app (in, out, args)    }}

To understand the bind function, you need to understand "closures" first. In the go language, the application manifests itself in a closed package. If you are familiar with functional programming, it is not difficult to find that this bind function is actually called the currying.

The pipe function is as follows:

Func pipe (    app1 func (in IO). Reader, out IO. Writer),    app2 func (in IO. Reader, out IO. Writer)) func (in IO. Reader, out IO. Writer) {    return func (in IO). Reader, out IO. Writer) {        PR, pw: = Io. Pipe ()        defer pw. Close ()        go func () {            defer pr. Close ()            app2 (PR, out)        } ()        App1 (in, PW)}    }

To understand the pipe function, you need to know the defer keyword and goroutine (go keyword) In addition to "closures". The defer statement executes when the function exits (regardless of whether an exception has occurred), and is typically used for cleanup operations of resources, such as closing file handles. With the defer statement, the error handling code in the Go language appears to be very elegant. Adding the Go keyword before a normal function call will cause the function to execute in parallel in the new goroutine. Understanding these backgrounds, this pipe function is not difficult to understand, is simply: first create a pipeline, let App1 read into the data (in), and to the writing end of the pipeline (PW) output, start a new goroutine, let App2 read from the pipeline read the data, and the processing results output (out). The app is a combination of App1 and APP2.

You can even combine multiple apps:

func pipe (Apps ... func (in IO). Reader, out IO. Writer)) func (in IO. Reader, out IO. Writer) {If len (apps) = = 0 {return nil} app: = Apps[0] for I: = 1; i < Len (apps); i++ {App1, app2: = App, apps[i] app = func (in IO. Reader, out IO. Writer) {PR, pw: = Io. Pipe () defer pw. Close () go func () {defer pr. Close () app2 (PR, out)} () App1 (in, PW)}} return app} 

Let's take a more practical example, assuming we have 2 applications tar (packaged), gzip (compressed):

• Func tar (IO. Reader, out IO. Writer, files []string]

• Func gzip (in IO. Reader, out IO. Writer)

Then the code that is packaged and compressed is:

Pipe (bind (tar, files), gzip) (nil, out)

Through the simulation of the pipe (pipeline) we can see that the go language is very powerful for parallel support, mainly thanks to the lightweight process of Go (goroutine).

An instance program to help understand the pipeline:

Package Mainimport ("IO" "OS" "Bufio" "bytes" "FMT" "StrConv")//bind functions are mainly used to integrate the pipe function, by turning the signature of the function into the desired appearance of the pipe Returns a function closure that passes a function literal app and string slice into which the Func bind (app func (in IO). Reader, out IO. Writer, args []string), args []string] func (in IO. Reader, out IO. Writer) {return func (in IO). Reader, out IO. Writer) {app (in, out, args)}}//two functions into the middle of the pipeline, the caller simply calls the function literal returned by the pipe, and passes in the end of the pipe, to implement the pipeline//return a new function closure func pipe (App1 func ( In IO. Reader, out IO. Writer), app2 func (in IO. Reader, out IO. Writer)) func (in IO. Reader, out IO. Writer) {return func (in IO). Reader, out IO. Writer) {PR, pw: = Io. Pipe () defer pw. Close () go func () {defer pr. Close () app2 (PR, out)} () App1 (in, PW)}}//reads each string of the args slice, takes it as a file name, reads the file, and adds the line number to the beginning of each file, writes to Out/ The main purpose is to guarantee the consistency of the pipe definition func app1 (in IO). Reader, out IO. Writer, args []string] {for _, V: = Range args {//fmt. Println (v) file, err: = OS. Open (v) if err! = Nil {Continue}defer file. Close () buf: = Bufio. Newreader (file) for i:=1;; I++{line, err: = buf. Readbytes (' \ n ') if err! = Nil {break}linenum: = StrConv. ItoA (i) nline: = []byte (LineNum + ") nline = append (nline, line ...) Out. Write (Nline)}}}

APP2 is mainly to convert bytes into uppercase, Chinese may be a bit of a problem, but mainly for demonstration, heavy in understanding the thought//read from, convert byte to Upper, write the result to Outfunc app2 (in IO . Reader, out IO. Writer) {rd: = Bufio. Newreader (in) P: = Make ([]byte, Ten) for {n, _: = Rd. Read (p) if n = = 0 {break}t: = bytes. ToUpper (P[:n]) out. Write (t)}}func Main () {args: = os. Args[1:]for _, V: = Range args {fmt. Println (v)}    P: = Pipe (Bind (App1, args), APP2) p (OS. Stdin, OS. Stdout)}