Translation Implementation of Go TCP socket

This is a creation in Article, where the information may have evolved or changed.

Original: TCP Socket implementation on Golang by Gian Giovani.

Note : The author does not analyze the implementation of the go socket from the source code level, but instead uses the strace tool to push back the behavior of the go socket. This approach can extend the means by which we analyze the code.
The source-level analysis can look at its implementation: NET poll, and some analysis articles: The Go Netpoller, the Go netpoller and timeout

The go language is my first choice for writing Web applications, and it hides a lot of detail, but still has the flexibility. The latest I used the Strace tool to analyze an HTTP program, purely hands-on, but still found some interesting things.

Here are strace the results:

1234567891011121314151617181920

% Time seconds Usecs/call calls Errors Syscall------ ----------- ----------- --------- --------- ---------------- the. - 0.397615 336 1185 in Futex 4. - 0.018009 3 7115 Clock_gettime 2. the 0.012735 + 654 epoll_wait 1. to 0.005701 6 911 Write 0. - 0.000878 3 335 Epoll_ctl 0. A 0.000525 1 915 457 Read 0. Geneva 0.000106 2 - Select 0. on 0.000059 0 the Close 0. on 0.000053 0 791 setsockopt 0. on 0.000035 0 158 Getpeername 0. on 0.000034 0 the Socket 0. on 0.000029 0 the getsockname 0. on 0.000026 0 159 getsockopt 0.xx 0.000000 0 7 Sched_yield 0.xx 0.000000 0 166 166 Connect 0.xx 0.000000 0 3 1 Accept4------ ----------- ----------- --------- --------- ---------------- -.xx 0.435805 12958 653 Total

There are a lot of interesting things in this analysis, but the number of read errors and the number of errors in the call are highlighted in this article futex .

At first I did not contemplate the invocation of Futex, most of it is nothing more than a wake-up call (wake called). Since this program handles hundreds of requests per second, it should contain a lot of go routine. On the other hand, it uses the channel, which also leads to a lot of block situations, so there are many Futex calls that are also normal. But then I found that this number also contains logic from other, followed by the table.

Why you no Read

Does anyone like the wrong (error)? There were hundreds of mistakes in just one minute, too bad, and that was the initial impression I had when I saw the results of this analysis. So read call what is it again?

123	Read("Get/xxx/v3?q=xx%20ch&d"..., 4096) = 520... read( 0 xc422aa4291, 1) = 1 eagain (Resource temporarily unavailable)

Each time the read invokes the same file descriptor, it is always (possibly) accompanied by an EAGAIN error. I remember this error, when the file descriptor is not ready for an operation, it will return this fault, the above example is the Operation read . The question is, why would go do that?

I guess this is probably epoll_wait a bug that provides the wrong event for each file descriptor ready ? Each of the file descriptors? It seems that the read event is twice times the error event, why is it twice times?

Frankly speaking, my epoll knowledge is very good, the program is just a simple processing event socket handler (similar). No multi-threading, no synchronization, very simple.

Through Google I found a great article to analyze the comments epoll , written by Marek.

An important summary of this article is that, for use in multi-threading epoll , unnecessary wakeup (wake up) is often unavoidable because we want to notify each worker waiting for an event.

This also explains the number of Futex wakes we have. Let's look at a simplified version to understand how to use it in an event-based socket handler epoll :

1. Bind socket listener to file descriptor , we call its_fd
2. Using epoll_create Create epoll file descriptor , we call ite_fd
3. epol_ctl s_fd e_fd Handle special events (usually) by bind to. EPOLLIN|EPOLLOUT
4. Creates an infinite loop (event loop), which is called in each loop to epoll_wait get the Ready connection
5. Handles ready connections, and notifies every worker in a multi-worker implementation

Using Strace I found that Golang using edge triggered Epoll
Using the strace edge triggered epoll I found Golang using:

1	Epoll_ctl (43, {epollin| Epollout| epollrdhup| Epollet, {u32=2490298448u64=1404908705506080

This means that the following procedure should be the implementation of the GO socket:

1,Kernel: Received a new connection.
2.Kernel: Notifies waiting threads threads A and B. Herd must wake up these two threads due to the "surprise Swarm" ("thundering Kernel") behavior of the level-triggered notification.
3,Thread A: Complete epoll_wait ().
4,Thread B: Complete epoll_wait ().
5,Thread A: Execute Accept (), success.
6.Thread B: Execute Accept (), fail, eagain error.

Now I have 80% certainty is this case, but let us use a simple program to analyze.

12345678910111213

Package mainimport"Net/http"func Main () {http. Handlefunc ("/", Handler) http. Handlefunc ("/test", Handler) http. Listenandserve (": 8080"nil)}func handler (w http. Responsewriter, R *http. Request) {}

A simple request after the strace result:

12345678

Epoll_wait (4, [{epollin| Epollout, {u32=2186919600, u64=140542106779312}}], -, -1) =1Futex (0x7c1bd8, Futex_wake,1) =1Futex (0X7C1B10, Futex_wake,1) =1Read5,"Get/http/1.1\r\nhost:localhost:"...,4096) =348Futex (0xc420060110, Futex_wake,1) =1Write5,"http/1.1 Ok\r\ndate:sat, J"..., the) = theFutex (0xc420060110, Futex_wake,1) =1Read5,0xc4200f6000,4096) = -1Eagain (Resource temporarily unavailable)

See epoll_wait There are two Futex calls that I think are worker executions as well as an error read.

If GOMAXPROCS set to 1, in single worker case:

12345678910111213

Epoll_wait (4, [{Epollin, {u32=1969377136, u64=140245536493424}}], -, -1) =1Futex (0x7c1bd8, Futex_wake,1) =1ACCEPT4 (3, {sa_family=af_inet6, sin6_port=htons (54400), Inet_pton (Af_inet6,":: ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [ -], sock_cloexec| Sock_nonblock) =6Epoll_ctl (4, Epoll_ctl_add,6, {epollin| Epollout| epollrdhup| Epollet, {u32=1969376752, u64=140245536493040}}) =0GetSockName (6, {sa_family=af_inet6, sin6_port=htons (8080), Inet_pton (Af_inet6,":: ffff:127.0.0.1", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [ -]) =0SetSockOpt6, Sol_tcp, Tcp_nodelay, [1],4) =0SetSockOpt6, Sol_socket, So_keepalive, [1],4) =0SetSockOpt6, Sol_tcp, TCP_KEEPINTVL, [ the],4) =0SetSockOpt6, Sol_tcp, Tcp_keepidle, [ the],4) =0ACCEPT4 (3,0xc42004db78,0xc42004db6c, sock_cloexec| Sock_nonblock) =-1Eagain (Resource temporarily unavailable) read (6,"Get/test?kjhkjhkjh Http/1.1\r\nho"...,4096) = theWrite6,"http/1.1 Ok\r\ndate:sat, J"...,139) =139Read6,"",4096)

When 1 worker,epoll_wait is used, only one futex wake up, and there is no error read. However, I find that this is not always the case, sometimes I can still get read error and two Futex wakeup.

And then.

In Marek's article, he spoke about Linux 4.5, which he could use EPOLLEXCLUSIVE . My Linux version is 4.8, why does the problem still occur? Maybe go does not use this flag, I hope that the future version can use this flag.

I learned a lot of knowledge from it, I hope you too.

[0] https://banu.com/blog/2/how-to-use-epoll-a-complete-example-in-c/
[1] https://idea.popcount.org/ 2017-02-20-epoll-is-fundamentally-broken-12/
[2] https://gist.github.com/wejick/ 2cef1f8799361318a62a59f6801eade8