Some thinking about socket API from Go

This is a creation in Article, where the information may have evolved or changed.

This article translated from: https://idea.popcount.org/2017-02-23-socket-api-thoughts/

A while ago, I wrote an article about select () system call development history. This article raises some interesting discussions.

The article and the discussion took a lot of my mind, but I finally understood what was the core issue:

is the UNIX process a CSP-style process? Is the file descriptor "channel" derived from the CSP? Is "select ()" equivalent to an ALT statement?

The direct answer is "no". CSPs are younger than Unix. As Tony Garnock-jones points out, the first article on CSP, published in 1978, has been a file descriptor since the early days of UNIX.

File descriptors are not channels, what should they be? The CSP channel is usually composable, and I think Unix's select is not. This is the 2nd:

A UNIX program that uses "select ()" does not use a combination. But maybe they can use it? can select () be implemented in a way that allows for composition?

When programming in Golang, you can use the ALT statement and the built-in channel. In general, Golang Channel is composable. Similar things in Erlang-you can avoid complex state machines by cleverly using the gen_server process.

Why can't the UNIX process model be this way? Why can't we combine UNIX processes in the same way as Golang or Erlang?

I would intuitively accuse the select system of calling, but the problem is more in-depth.

Buffering Agent

We can't give up select? Can we do without it? In the previous article we mentioned two examples:

• inet Daemon Process

• Console multiplexing

There are many problems that cannot be separated from select. David Wragg referred to the "buffering agent" issue:

Imagine a process that is buffering. Consider a simple program that reads data from one stream, buffers it, and writes another stream. If you use a pipeline between them to split it into two processes, you will be constrained by the size of the kernel pipe buffer. In order to have an arbitrary buffer size, it must make a select operation for the readability of the input stream and the writable nature of the output stream in a single process.

The same problem was mentioned in Chris Benman's 2010 blog.

Memory consumption for idle connections

Earlier this year, Chris mentioned another issue that required select: "Memory consumption for idle connections":

• One drawback of the queued IO model is the memory consumption of idle connections

This post is a follow-up to Evan Klitzke's article:

• Goroutines, non-blocking I/O and memory usage

• Related discussions on YCombinator

This memory consumption argument is presented in the Golang bug tracker:

NET: Adding a wait mechanism on Tcpconn

To me, this sounds like an introduction to the callback hell.

BSD Socket API Retrofit

Finally, Martin Sustrick wrote this RFC proposal for the pure blocking sockets API:

• BSD Socket API Retrofit

• YCombinator Discussion

This article does not solve the problem by pushing the select issue to the next level:

The memo assumes that a valid concurrency implementation already exists, that it takes up to hundreds of nanoseconds to create a new lightweight process, and that context switching requires dozens of nanoseconds. Note that such a concurrency system has already been deployed. A famous example is Golang's goroutine, but there are other available.

I really like the idea of just exposing the blocking API to the user.

Final idea

Although the UNIX socket model and CSP have existed for nearly 40 years, they are not individually closed. There are many new ideas in this area: optimizing memory usage, finding the right composable abstraction, and returning to exposing the pure blocking API.