TCP/IP network programming in the Go language

This is a creation in Article, where the information may have evolved or changed.

Go language TCP/IP network programming

At first glance, connecting two processes through the TCP/IP layer can be scary, but in the go language it may be much simpler than you think.

Application scenarios for sending data at the TCP/IP layer

Of course, in many cases, not most cases, the use of higher-level network protocols will undoubtedly be better, because you can use the gorgeous API, they hide a lot of technical details. Now according to different needs, there are many options, such as Message Queuing protocol, GRPC, Protobuf, Flatbuffers, RESTful website API, WebSocket and so on.

However, in some special scenarios, especially for small projects, choosing any other way will feel bloated, not to mention the need to introduce additional dependency packages.

Fortunately, creating simple network traffic using the standard library's net package is no more difficult than what you see.

Because the go language has the following two points simplified.

Simplified 1: connection is IO stream

Net. The Conn interface implements the Io.reader, Io. Writer and Io.closer interface. Therefore, the TCP connection can be treated like any other IO stream.

You might think, "Well, I can send string or byte shards in TCP, very good, but what about complex data structures?" For example, are we encountering data of struct type? "

Simplifying the 2:go language knowing how to decode complex types effectively

When it comes to sending coded structured data over the network, the first thing to think about is JSON. But wait a minute.-The standard library Encoding/gob package for the Go language provides a way to serialize and send serial session go data types without having to add string tags to structs, go language-incompatible JSON, or wait for JSON to be used. Unmarshal to parse the text into binary data in a strenuous effort.

GOB encoding and decoding can directly manipulate the IO stream, which perfectly matches the first simplification.

Let's implement a simple app together with these two simplified rules.

The goal of this simple app

This app should do two things:

• Sends and receives a simple string message.
• The structure is sent and received through GOB.

The first part, sending a simple string, will demonstrate how easy it is to send data over a TCP/IP network without the use of advanced protocols.
The second part, a little bit deeper, sends a complete structure over the network, which uses strings, shards, mappings, and even recursive pointers that are contained in itself.

Sin has gob bag, to do these effortless.

Client server

Structure to be sent Structure after decoding
testStruct structure testStruct structure
    | ^
    V |
gob encoding ----------------------------> gob decoding
    | ^
    V |
   Send ============= Network ================= Receive
Basic elements of sending string data over TCP
Sender
Sending a string requires three simple steps:

Open the connection corresponding to the receiving process.
Write a string.
Close the connection.
The net package provides a pair of methods to achieve this functionality.

ResolveTCPAddr (): This function returns the TCP terminal address.
DialTCP (): Similar to dial-up for TCP networks.
Both methods are defined in the src / net / tcpsock.go file of the go source code.

func ResolveTCPAddr (network, address string) (* TCPAddr, error) {
    switch network {
    case "tcp", "tcp4", "tcp6":
    case "": // a hint wildcard for Go 1.0 undocumented behavior
        network = "tcp"
    default:
        return nil, UnknownNetworkError (network)
    }
    addrs, err: = DefaultResolver.internetAddrList (context.Background (), network, address)
    if err! = nil {
        return nil, err
    }
    return addrs.forResolve (network, address). (* TCPAddr), nil
}
ResolveTCPAddr () takes two string parameters.

network: must be a TCP network name, such as tcp, tcp4, tcp6.
address: TCP address string. If it is not a literal IP address or the port number is not a literal port number, ResolveTCPAddr resolves the incoming address to the address of the TCP terminal. Otherwise, pass in a pair of literal IP address and port number as the address. The address parameter can use the host name, but this is not recommended because it returns at most one IP address of the host name.
ResolveTCPAddr () receives a string representing the TCP address (for example, localhost: 80, 127.0.0.1:80, or [:: 1]: 80, all representing port 80 of the machine), returns (net.TCPAddr pointer, nil ) (If the string cannot be parsed into a valid TCP address, (nil, error) is returned).

func DialTCP (network string, laddr, raddr * TCPAddr) (* TCPConn, error) {
    switch network {
    case "tcp", "tcp4", "tcp6":
    default:
        return nil, & OpError {Op: "dial", Net: network, Source: laddr.opAddr (), Addr: raddr.opAddr (), Err: UnknownNetworkError (network)}
    }
    if raddr == nil {
        return nil, & OpError {Op: "dial", Net: network, Source: laddr.opAddr (), Addr: nil, Err: errMissingAddress}
    }
    c, err: = dialTCP (context.Background (), network, laddr, raddr)
    if err! = nil {
        return nil, & OpError {Op: "dial", Net: network, Source: laddr.opAddr (), Addr: raddr.opAddr (), Err: err}
    }
    return c, nil
}
The DialTCP () function receives three parameters:

network: This parameter is the same as the network parameter of ResolveTCPAddr. It must be the TCP network name.
laddr: a pointer of type TCPAddr, representing the local TCP address.
raddr: pointer of TCPAddr type, which represents the remote TCP address.
It will connect and dial two TCP addresses and return this connection as a net.TCPConn object (error returned if connection fails). If we don't need too much control over the Dial setting, then we can use Dial () instead.

func Dial (network, address string) (Conn, error) {
    var d Dialer
    return d.Dial (network, address)
}
The Dial () function receives a TCP address and returns a normal net.Conn. This is enough for our test cases. However if you need functionality available only on TCP connections, you can use TCP variants (DialTCP, TCPConn, TCPAddr, etc.).

After successfully dialing, we can treat the new connection as other input and output streams as described above. We can even wrap the connection into bufio.ReadWriter so that we can use various ReadWriter methods, such as ReadString (), ReadBytes, WriteString, etc.

func Open (addr string) (* bufio.ReadWriter, error) {
    conn, err: = net.Dial ("tcp", addr)
    if err! = nil {
        return nil, errors.Wrap (err, "Dialing" + addr + "failed")
    }
    // Wrap net.Conn object into bufio.ReadWriter
    return bufio.NewReadWriter (bufio.NewReader (conn), bufio.NewWriter (conn)), nil
}
Remember that the buffer Writer needs to call the Flush () method after writing, so that all data will be flushed to the underlying network connection.
Finally, each connection object has a Close () method to terminate the communication.

Fine tuning
The Dialer structure is defined as follows:

type Dialer struct {
    Timeout time.Duration
    Deadline time.Time
    LocalAddr Addr
    DualStack bool
    FallbackDelay time.Duration
    KeepAlive time.Duration
    Resolver * Resolver
    Cancel <-chan struct {}
}
Timeout: The maximum amount of time dialing waits for the connection to end. If Deadline is also set, you can fail earlier. There is no timeout by default. When using TCP and dialing hostnames with multiple IP addresses, the timeout is divided between them. With or without a timeout, the operating system can force an earlier timeout. For example, the TCP timeout is generally around 3 minutes.
Deadline: An absolute point in time when dialing is about to fail. If Timeout is set, it may fail earlier. A value of 0 means that there is no deadline, or that it depends on the operating system or uses the Timeout option.
LocalAddr: is the local address used when dialing an address. This address must be of a fully compatible type for the network address to be dialed. If nil, a local address is automatically selected.
DualStack: This attribute enables RFC 6555 compatible "Happy Eyeballs" dialing. When the network is tcp, the host in the address parameter can be resolved into IPv4 and IPv6 addresses. This allows the client to tolerate the network rules of an address family a bit.
FallbackDelay: When DualStack is enabled, specifies the amount of time to wait before generating a fallback connection. If set to 0, the default is 300ms.
KeepAlive: Specify the time to keep alive for active network connections. If set to 0, keep-alive is not enabled. Network protocols that do not support keep-alive will ignore this field.
Resolver: Optional, specifies an alternative resolver to use.
Cancel: Optional channel. Its closure indicates that dialing should be canceled. Not all dial types support dial cancellation. Deprecated. Use DialContext instead.
There are two options available for fine-tuning.

Therefore the Dialer interface provides two options that can be fine-tuned:

DeadLine and Timeout options: Timeout settings for unsuccessful dialing.
KeepAlive option: Manage the life span of the connection.
type Conn interface {
    Read (b [] byte) (n int, err error)
    Write (b [] byte) (n int, err error)
    Close () error
    LocalAddr () Addr
    RemoteAddr () Addr
    SetDeadline (t time.Time) error
    SetReadDeadline (t time.Time) error
    SetWriteDeadline (t time.Time) error
}
The net.Conn interface is a stream-oriented general network connection. It has the following interface methods:

Read (): Read data from the connection.
Write (): Write data to the connection.
Close (): Close the connection.
LocalAddr (): Returns the local network address.
RemoteAddr (): Returns the remote network address.
SetDeadline (): Set the read and write deadlines related to the connection. Equivalent to calling SetReadDeadline () and SetWriteDeadline () at the same time.
SetReadDeadline (): Sets the timeout deadline for future read calls and currently blocked read calls.
SetWriteDeadline (): Sets the timeout deadline for future write calls and currently blocked write calls.
The Conn interface also has deadline settings; there are (SetDeadLine ()) for the entire connection, and (SetReadDeadLine () and SetWriteDeadLine ()) for specific read and write calls.

Note that deadline is a (wallclock) time fixed point. Unlike timeout, they are not reset after a new event. So every activity on the connection must set a new deadline.

The sample code below does not use deadline, because it is simple enough that we can easily see when it will get stuck. When Ctrl-C, we manually trigger the deadline tool.

On the receiving end
The receiving steps are as follows:

Open listening on the local port.
Spawn g when the request comes oroutine to process the request.
In goroutine, read the data. You can also optionally send a response.
Close the connection.
Listening needs to specify the port number for local listening. In general, the listening application (also called server) announces the port number to listen on. If a standard service is provided, the relevant port corresponding to this service is used. For example, web services usually listen on 80 to serve HTTP and port 443 to serve HTTPS requests. The SSH daemon listens on port 22 by default, and the WHOIS service uses port 43.

type Listener interface {
    // Accept waits for and returns the next connection to the listener.
    Accept () (Conn, error)

    // Close closes the listener.
    // Any blocked Accept operations will be unblocked and return errors.
    Close () error

    // Addr returns the listener's network address.
    Addr () Addr
}
func Listen (network, address string) (Listener, error) {
    addrs, err: = DefaultResolver.resolveAddrList (context.Background (), "listen", network, address, nil)
    if err! = nil {
        return nil, & OpError {Op: "listen", Net: network, Source: nil, Addr: nil, Err: err}
    }
    var l Listener
    switch la: = addrs.first (isIPv4). (type) {
    case * TCPAddr:
        l, err = ListenTCP (network, la)
    case * UnixAddr:
        l, err = ListenUnix (network, la)
    default:
        return nil, & OpError {Op: "listen", Net: network, Source: nil, Addr: la, Err: & AddrError {Err: "unexpected address type", Addr: address}}
    }
    if err! = nil {
        return nil, err // l is non-nil interface containing nil pointer
    }
    return l, nil
}
The core part of the net package to implement the server is:

net.Listen () creates a new listener on the given local network address. If only the port number is passed to it, such as ": 61000", the listener will listen on all available network interfaces. This is convenient because computers usually provide at least two active interfaces, a loopback interface, and at least one real network card. This function returns Listener if successful.

The Listener interface has an Accept () method to wait for a request to come in. It then accepts the request and returns a new connection to the caller. Accept () is generally called in a loop and can serve multiple connections simultaneously. Each connection can be handled by a separate goroutine, as shown in the following code.

Code part
Rather than having the code push some bytes back and forth, I want it to demonstrate something more useful. I want it to send different commands to the server with different data carriers. The server should be able to identify each command and decode the command data.

The client in our code sends two types of commands: "STRING" and "GOB". They all terminate with a newline character.

The "STRING" command contains one line of string data, which can be processed by simple read and write operations in bufio.

The "GOB" command consists of a structure that contains fields, including a slice and a map, and even a pointer to itself. As you can see, when running this code, it is not surprising that gob packages can move this data through our network connection.

We basically have some ad-hoc protocol (ad-hoc protocol: ad hoc, special purpose, ad hoc, ad hoc), both the client and server follow it. The command line is followed by a newline and then data. For each command, the server must know the exact format of the data and know how to handle it.

To achieve this, the server code takes a two-step approach.

Step 1: When the Listen () function receives a new connection, it will generate a new goroutine to call handleMessage (). This function reads the command name from the connection, queries the appropriate handler function from the map, and then calls it.
Step 2: The selected processor function reads and processes the command line data.
package main

import (
    "bufio"
    "encoding / gob"
    "flag"
    "github.com/pkg/errors"
    "io"
    "log"
    "net"
    "strconv"
    "strings"
    "sync"
)

type complexData struct {
    N int
    S string
    M map [string] int
    P [] byte
    C * complexData
}

const (
    Port = ": 61000"
)
Outcoing connections
Using a launch connection is a snapshot. net.Conn meets the io.Reader and io.Writer interfaces, so we can treat TCP connections like any other Reader and Writer.

func Open (addr string) (* bufio.ReadWriter, error) {
    log.Println ("Dial" + addr)
    conn, err: = net.Dial ("tcp", addr)

    if err! = nil {
        return nil, errors.Wrap (err, "Dialing" + addr + "failed")
    }

    return bufio.NewReadWriter (bufio.NewReader (conn), bufio.NewWriter (conn)), nil
}
Open a TCP address connection. It returns a TCP connection with a timeout and wraps it into a buffered ReadWriter. Dial the remote process. Note that the local port is allocated on the fly. If you must specify a local port number, use the DialTCP () method.

Incoming connection
This section is a bit related to the preparation of incoming data. According to the ad-hoc protocol we introduced earlier, command name + newline character + data + newline character. Natural data is related to specific commands. To handle this situation, we create an Endpoint object with the following properties:

It allows to register one or more handler functions, each of which can process a special command.
It dispatches specific commands to related processor functions based on the command name.
First we declare a HandleFunc type, which is a function type that receives a bufio.ReadWriter pointer value, which is the handler function we will register for each different command later. The parameters it receives are net.Conn connections wrapped using the ReadWriter interface.

type HandleFunc func (* bufio.ReadWriter)
Then we declare an Endpoint structure type, which has three properties:

listener: The Listener object returned by net.Listen ().
handler: A map used to hold registered handler functions.
m: a mutex lock, used to solve the multi-goroutine insecurity problem of map.
type Endpoint struct {
    listener net.Listener
    handler map [string] HandleFunc
    m sync.RWMutex // Maps are not thread-safe, so a mutex is required to control access.
}

func NewEndpoint () * Endpoint {
    return & Endpoint {
        handler: map [string] HandleFunc {},
    }
}

func (e * Endpoint) AddHandleFunc (name string, f HandleFunc) {
    e.m.Lock ()
    e.handler [name] = f
    e.m.Unlock ()
}

func (e * Endpoint) Listen () error {
    var err error
    e.listener, err = net.Listen ("tcp", Port)
    if err! = nil {
        return errors.Wrap (err, "Unable to listen on" + e.listener.Addr (). String () + "\ n")
    }
    log.Println ("Listen on", e.listener.Addr (). String ())
    for {
        log.Println ("Accept a connection request.")
        conn, err: = e.listener.Accept ()
        if err! = nil {
            log.Println ("Failed accepting a connection request:", err)
            continue
        }
        log.Println ("Handle incoming messages.")
        go e.handleMessages (conn)
    }
}

// handleMessages reads the first newline connected to it. Based on this string, it will call the appropriate HandleFunc.

func (e * Endpoint) handleMessages (conn net.Conn) {
    // Wrap the connection to a buffered reader for easy reading
    rw: = bufio.NewReadWrite (bufio.NewReader (conn), bufio.NewWriter (conn))
    defer conn.Close ()

    // Read from the connection until EOF is encountered. Expect the next input to be the command name. Call the registered processor for the command.

    for {
        log.Print ("Receive command '")
        cmd, err: = rw.ReadString ('\ n')
        switch {
        case err == io.EOF:
            log.Println ("Reached EOF-close this connection. \ n ---")
            return
        case err! = nil:
            log.Println ("\ nError reading command. Got: '" + cmd + "' \ n", err)
        }

        // Trim extra carriage returns and spaces in the request string-ReadString will not strip any newlines.

        cmd = strings.Trim (cmd, "\ n ")
        log.Println (cmd + "'")

        // Get the appropriate handler function from the handler map and call it.

        e.m.Lock ()
        handleCommand, ok: = e.handler [cmd]
        e.m.Unlock ()

        if! ok {
            log.Println ("Command '" + cmd + "' is not registered.")
            return
        }

        handleCommand (rw)
    }
}
The NewEndpoint () function is a factory function for Endpoint. It only initializes the handler mapping. To simplify the problem, suppose the port that our terminal listens to is fixed.

The Endpoint type declares several methods:

AddHandleFunc (): Use a mutex to safely add handler functions for the handler attribute to handle specific types of commands.
Listen (): Start listening on all interfaces of the terminal port. Before calling Listen, you must register at least one handler function via AddHandleFunc ().
HandleMessages (): Wrap the connection in bufio, then read it in two steps. First read the command plus newline, we get the command name. Then the handler function corresponding to the registered command is obtained through the handler, and then this function is dispatched to perform data reading and parsing.
Notice how the dynamic functions are used above. Find a specific function according to the command name, and then assign this specific function to handleCommand. In fact, the variable type is HandleFunc, which is the type of the processor function declared earlier.
You need to register at least one handler function before calling Endpoint's Listen method. So we define two types of handler functions below: handleStrings and handleGob.

The handleStrings () function receives and processes handler functions that send only string data in our instant protocol. The handleGob () function is a complex structure that receives and processes the sent gob data. handleGob is a little more complicated. In addition to reading data, we need to decode the data.

We can see that rw.ReadString ('n') is used twice to read the string. When it encounters a newline, it stops and saves the read content to the string. Note that this string contains a trailing newline.

In addition, for ordinary string data, we directly use Buffio to wrap the connected ReadString to read it. For complex gob structures, we use gob to decode the data.

func handleStrings (rw * bufio.ReadWriter) {
    log.Print ("Receive STRING message:")
    s, err: = rw.ReadString ('\ n')
    if err! = nil {
        log.Println ("Cannot read from connection. \ n", err)
    }

    s = strings.Trim (s, "\ n")
    log.Println (s)

    -, err = rw.WriteString ("Thank you. \ n")
    if err! = nil {
        log.Println ("Cannot write to connection. \ n", err)
    }

    err = rw.Flush ()
    if err! = nil {
        log.Println ("Flush failed.", err)
    }
}

func handleGob (rw * bufio.ReadWriter) {
    log.Print ("Receive GOB data:")
    var data complexData
     
    dec: = gob.NewDecoder (rw)
    err: = dec.Decode (& data)

    if err! = nil {
        log.Println ("Error decoding GOB data:", err)
        return
    }

    log.Printf ("Outer complexData struct: \ n% # v \ n", data)
    log.Printf ("Inner complexData struct: \ n% # v \ n", data.C)
}
Client and server functions
With everything in place, we can prepare our client and server functions.

The client function connects to the server and sends STRING and GOB requests.
The server starts listening for requests and triggers the appropriate handler.
// called when the application uses -connect = ip address

func client (ip string) error {
    testStruct: = complexData {
        N: 23,
        S: "string data",
        M: map [string] int {"one": 1, "two": 2, "three": 3},
        P: [] byte ("abc"),
        C: & complexData {
            N: 256,
            S: "Recursive structs? Piece of cake!",
            M: Map [string] int {"01": "10": 2, "11": 3},
        },
    }

    rw, err: = Open (ip + Port)
    if err! = nil {
        return errors.Wrap (err, "Client: Failed to open connection to" + ip + Port)
    }

    log.Println ("Send the string request.")

    n, err: = rw.WriteString ("STRING \ n")
    if err! = nil {
        return errors.Wrap (err, "Could not send the STRING request (" + strconv.Itoa (n) + "bytes written)")
    }

    // Send a STRING request. Send the request name and send the data.

    log.Println ("Send the string request.")

    n, err = rw.WriteString ("Additional data. \ n")
    if err! = nil {
        return errors.Wrap (err, "Could not send additional STRING data (" + strconv.Itoa (n) + "bytes written)")
    }

    log.Println ("Flush the buffer.")
    err = rw.Flush ()
    if err! = nil {
        return errors.Wrap (err, "Flush failed.")
    }

    // read the response

    log.Println ("Read the reply.")

    response, err: = rw.ReadString ('\ n')
    if err! = nil {
        return errors.Wrap (err, "Client: Failed to read the reply: '" + response + "'")
    }

    log.Println ("STRING request: got a response:", response)
   
    // Send a GOB request. Create an encoder to convert it directly to the request name of rw.Send. Send GOB

    log.Println ("Send a struct as GOB:")
    log.Printf ("Outer complexData struct: \ n% # v \ n", testStruct)
    log.Printf ("Inner complexData struct: \ n% # v \ n", testStruct.C)
    enc: = gob.NewDecoder (rw)
    n, err = rw.WriteString ("GOB \ n")
    if err! = nil {
        return errors.Wrap (err, "Could not write GOB data (" + strconv.Itoa (n) + "bytes written)")
    }

    err = enc.Encode (testStruct)
    if err! = nil {
        return errors.Wrap (err, "Encode failed for struct:% # v", testStruct)
    }

    err = rw.Flush ()
    if err! = nil {
        return errors.Wrap (err, "Flush failed.")
    }

    return nil
}
The client function is executed when the connect flag is specified when the application is executed. This code can be seen later.

The following is the server program server. The server listens for incoming requests and dispatches them to the registered specific handlers according to the request command name.

func server () error {
    endpoint: = NewEndpoint ()

    // add handler function

    endpoint.AddHandleFunc ("STRING", handleStrings)
    endpoint.AddHandleFunc ("GOB", handleGOB)

    // start listening

    return endpoint.Listen ()
}
main function
The following main function can start both the client and the server, depending on whether the connect flag is set. If this flag is not present, the server starts the process and listens for incoming requests. If there is a flag, start as a client and connect to the host specified by this flag.

You can run both processes on the same machine using localhost or 127.0.0.1.

func main () {
    connect: = flag.String ("connect", "", "IP address of process to join. If empty, go into the listen mode.")
    flag.Parse ()

    // If the connect flag is set, enter client mode

    if * connect! = '' {
        err: = client (* connect)
        if err! = nil {
            log.Println ("Error:", errors.WithStack (err))
        }
        log.Println ("Client done.")
        return
    }

    // Otherwise enter server mode

    err: = server ()
    if err! = nil {
        log.Println ("Error:", errors.WithStack (err))
    }

    log.Println ("Server done.")
}

// Set field flags for logging

func init () {
    log.SetFlags (log.Lshortfile)
}
How to get and run the code
Step 1: Get the code. Note that the -d flag automatically installs binaries into the $ GOPATH / bin directory.

go get -d github.com/appliedgo/networking
Step 2: cd to the source code directory.
cd $ GOPATH / src / github.com / appliedgo / networking

Step 3: Run the server.
go run networking.go

Step 4: Open another shell, also go to the source directory (step 2), and then run the client.
go run networking.go -connect localhost

Tips
If you want to modify the source code slightly, here are some suggestions:

Run the client and server on different machines (in the same LAN).
Enhance complexData with more maps and pointers, and see how gob cope with it.
Start multiple clients at the same time and see if the server can handle them.
2017-02-09: Maps are not thread-safe, so if you use the same map in different goroutines, you should use a mutex to control map access.
In the above code, the map is added before the goroutine is started, so you can safely modify the code and call AddHandleFunc () when the handleMessages goroutine is already running.
Summary of the knowledge learned in this chapter
---- 2018-05-04 -----

bufio application.
gob application.
Maps are not safe when shared between multiple goroutines.
Reference link
Original TCP / IP Network
Simple TCP server and client
gob data