[translation] the gob of flying

This is a creation in Article, where the information may have evolved or changed.

The original text of the topic is called "Gobs on The Wire", the author cleverly used the word "gob". Gob was originally a packet of Golang on the network communication protocol. And here, I feel that the title can also be translated as "about the big rotor on the line ...". Well, I have to admit, this translation is really indecent.

———— Translation Split Line ————

The gob of flying

This week, I want to talk to you. How to write a Client/server system based on synchronous request and asynchronous event notification.

To help learn Go, I cloned a conserver command line service. Of course, there is no need for another command-line service for the world now. However, because the go language gives developers the features that are ideal for command line services, this can be a very interesting experience. The task of the command-line service is to summarize the output from one or more serial ports (or one or more of the various systems that implement the Rtelnet protocol using the TCP connection). It logs the output to a file, while allowing the output to be seen by others in real time. It allows only one user to read and write to actually control the device.

The go version of the command-line service is called Gocons. Take a look first and then come back here. As always, I'll start by saying that I learned something about go when I was writing this.

Initially, I tried to use Netchan to build this. I think if the client requests to the server on a channel, the same as the same, it would be a cool thing to receive a notification when an event occurs on a command line. However, because Netchan cannot handle multiple channels, it crashes completely. In my opinion, it is theoretically possible to have Netchan handle multiple channels in some way, but it does look very difficult, and multiple channels may never be synchronized. So I turned to the right direction ...

The next step is to use the RPC package to implement the protocol between the client and the server. However, this brings up another problem: RPC, which is defined synchronously. It does not explicitly define how the server side uses RPC packets to issue asynchronous events to the client, such as "These bytes have arrived." I need to write the protocol myself, through two steps, a synchronous call, such as "Can I listen to this?" Yes. "and Asynchronous events" The command line you are monitoring has just output these bytes ". I've written these protocols before, and it's not hard, it's just a little cumbersome. And the GOB package that go provides is all you need to develop a protocol.

And all you have to do is draw up the protocol message and let GOB handle the package and package, which is to compute the separation of the messages. In our case, this type is called Connreq and connreply. In an ideal world, these should be the public types of a library, which both the client and server side use. In Gocons, I lazy, just copied and pasted them. The client is in net. There are gob.decode on the tcpconn, and the result is connreq (if not, it means something is wrong, the client can kill the connection or decode the subsequent content on the connection). Since go does not have a union (non-type-safe), Connreq and connreply should include all fields if the given protocol message is not used. I did not think about this protocol, but since the unused field might be a byte segment or a string, there could be many, and an empty byte segment would be encoded as nil instead of the entire byte buffer filled with 0.

The more elaborate idea is to construct a hierarchical type, the simplest type (with only one int named type) as the basis, followed by a complex type. But make it clear to gob. What type of decode is very difficult; it's like you have to split the protocol into two parts and call go on it separately. Encode. The first one tells you what type it is, and the second gob can be a phrase that contains data. Anyway, I'm not going to do that in gocons. Simple is good!

In the server, there are two snippets of code that are interesting. One is using JSON as the configuration file format. The other is how to send new data to all listeners.

The first one is relatively simple. Just a demo without using JSON. Unmarshall, how to get data from a JSON file. I can't figure out json. Unmarshall, so in case I try without it, let JSON. Decode work. I'm not saying this is good, but it works, which may help others who are looking for examples in the go reading JSON.

The desired input is this:

{"Consoles": {"Firewall": "ts.company.com:2070", "web01": "ts.company.com:2071", "web02": "Ts.co mpany.com:2072 "," web03 ":" Ts.company.com:2073 "}}

The goal is to call Addconsole for each key value that corresponds to the consoles.

This is done if you don't want to (or know how to) use JSON. Unmarshal Bar:

  R, Err: = OS. Open (*config, OS. O_rdonly, 0)   If err! = Nil {    log. EXITF ("Cannot read config file%v:%v", *config, err)  }  dec: = json. Newdecoder (R)   var conf interface{}  err = Dec. Decode (&conf)   If err! = Nil {    log. Exit ("JSON decode:", err)  }  hash, OK: = Conf. (map[string]interface{})   If!ok {    log. Exit ("JSON format Error:got%T", conf)  }  consoles, OK: = hash["Consoles"]  if!ok {    log. Exit ("JSON format Error:key consoles not found")  }  c2, OK: = consoles. (map[string]interface{})   If!ok {    log. EXITF ("JSON format error:consoles key wrong type,%T", consoles)  }  for k, V: = range C2 {    S, OK : = V. (string)     If OK {      Addconsole (k, s)    } else {      log. Exit ("Dial string for console%v are not a string.", K)    } }

The pattern here is roughly the same, JSON. Decode provides a interface{}, then uses the type selector based on the structure and then obtains what you expect to get there.

A more straightforward approach is to use JSON. Unmarshal. It's hard to understand how to use it from the documentation, but fortunately this article makes it look clearer.

A server is a series of goroutine components that process I/O in a loop. Each command line it monitors has a read goroutine and a write Goroutine. Read from which to get the bytes, and then distribute to all listening gocons clients. It manages a list of lists that contain clients, but another data structure might work better. Whether it's in net. Tcpconn or channels, the client is not sorted. The channel waiting for the new data is like the proxy goroutine of the client. When each client connects, it creates a pair of goroutine, one for reading and one for writing. This allows us to implement blocking reads on the input (see Dec for an example.) Decode) without worrying about blocking other tasks on the server.

A separate Goroutine guarantee is responsible for writing to the TCP connection so that no locks can be used. As an exercise, you can have multiple command-line managers say, "I have some data that requires the multiplexing of TCP connections!" "Without having to worry about them interfering with each other when writing data to the connection. (The current implementation can only listen to one command line at a time.) ）

The following fragment shows how to package and send notifications to all command-line viewers when new content is available:

    Select {     //a bunch of other channels to monitor here...      Case data: = <-m.dataCh:        if closed (M.datach) {          break l    & nbsp  }       //multicast the data to the listeners        for L: = listeners; L! = nil;  L = l.next {          if closed (l.ch) {           //Todo:need to  Remove this node from the list, not just Mark nil            l.ch = nil            Log. Print ("Marking listener", L, "no longer active.")          }          if l.ch! = Nil {            OK: = l.ch <-consoleevent{data}            If!ok {        &N Bsp     Log. Print ("Listener", L, "lost aN event. ")            }         }       }

In this way, we set up a new consoleevent and send it to each listener. This is a bit wasteful: it produces a lot of garbage, which means that the garbage collector needs to work harder. You can create a consoleevent and then send this one to all listeners. However, if you share memory like this, you need the developer to decide whether to make the shared memory read-only, or to use mutexes to control its access. In our case, we used a read-only approach like this:

New event arrived from the console manager, so send it down the TCP connection case EV: = <-evch:reply.code = Data Reply.data = Ev.data err: = Enc. Encode (Reply) if err! = Nil {log. Print ("Connection", C.RW, ", failed to send data:", err) Break L}

In this mode, two goroutine are only responsible for reading and writing, which is like magic. This is a fantastic reduction in the amount of code needed to implement Gocons. The original command-line service required hundreds of lines of complex code, about setting the Select Mask, waiting for SELECT, detecting if the FD needed accept () or read (), or something else (and finding the correct data structure to make FD available). In Gocons, as well as other go programs, such as HTTP packages implemented by HTTP services, you can use blocking read, so that go is scheduled to run so that the entire system is not blocked.

However, it would be interesting to consider the case when the write to the client TCP connection is blocked. When the system is blocking on write, it eventually discards and blocks all other clients from being read from the command line. In order to do this, you need to build firewalls where appropriate: shared resources should not let individuals block them. You need to set up a queue between the client proxy goroutine and the command line manager to read the channel between Goroutine, let it implement non-blocking write on the channel, and handle it when one is blocked. For example, you can close the channel and say, "Hey, your drainage is not smooth, you should clean it up and then come to me." ”

Using go to write and debug this server, let me learn a lot of things. And I still have a lot of things to learn: there are still some mysterious things in the code, such as why I need the runtime. Gosched () guarantees that there will be no blocking and how to handle the trouble of closing the channel in select. There are more mysterious work hidden in SetOwner, first and foremost: how to find bugs in "pump goroutine" that will be forwarded from one place to another (in the Go Runtime environment, or I can understand it)