Linux Latest So_reuseport Features

1. Preface

Yesterday summed up the Linux under the network programming "surprise group" phenomenon, give the method of Nginx processing surprise group, using mutual exclusion lock. As an example of the advantages of multi-core, the current network programming model is multi-process or multi-threading, according to the location of Accpet, divided into the following scenarios:

(1) A single process or thread creates a socket and makes a listen and accept, receives a connection after the process is created and the thread processes the connection

(2) Single process or thread create socket, and listen, pre-create a lot of worker process or thread accept () in the same server socket,

Both of these models give full play to the benefits of multicore CPUs, although they can be threaded and CPU-bound, but they all exist:

• Single listener worker threads can become a bottleneck in high-speed connection access processing
• Competing between multiple threads get service sockets
• Cache line Jumps
• Hard to load balance between CPUs
• As the number of cores expands, performance does not increase with

Reference: http://www.blogjava.net/yongboy/archive/2015/02/12/422893.html

Linux Kernel 3.9 brings the So_reuseport feature to solve most of the above problems.

2, So_reuseport solved what problem

So_reuseport supports multiple processes or threads bound to the same port, improving the performance of the server program and solving the problem:

• Allow multiple sockets bind ()/listen () the same TCP/UDP port
  • Each thread has its own server socket
  • There is no competition for locks on server sockets
• Load balancing at the kernel level
• Security level, sockets that listen to the same port can only be located under the same user

Its core implementation is mainly three points:

• Expand socket option to add the So_reuseport option to set Reuseport.
• Modify the BIND system call implementation so that support can be bound to the same IP and port
• Modifies the implementation of the new connection, and when looking for listener, can support a balanced selection between multiple sock listening to the same IP and port.

With So_resueport, each process can create its own socket, bind, listen, accept the same address and port, respectively, are independent and equal. Let the multi-process listen to the same port, the process accept socket fd is different, when a new connection is established, the kernel will only wake up a process accept , and ensure the equalization of wake.

3. Test code

1Include <stdio.h>2#include <unistd.h>3#include <sys/types.h>4#include <sys/socket.h>5#include <netinet/inch.h>6#include <arpa/inet.h>7#include <assert.h>8#include <sys/wait.h>9#include <string.h>Ten#include <errno.h> One#include <stdlib.h> A#include <fcntl.h> -  - #defineIP "127.0.0.1" the #definePORT 8888 - #defineWORKER 4 - #defineMAXLINE 4096 -  + intWorkerinti) - { +     structsockaddr_in address;  ABzero (&address,sizeof(address));  ataddress.sin_family =af_inet;  -Inet_pton (Af_inet, IP, &address.sin_addr);  -Address.sin_port =htons (PORT);  -  -     intLISTENFD = socket (pf_inet, Sock_stream,0);  -ASSERT (LISTENFD >=0);  in  -     intval =1; to     /*Set So_reuseport*/ +     if (setsockopt (LISTENFD, Sol_socket, So_reuseport, &val, sizeof (val)) <0) { -Perror ("setsockopt ()");  the     }     *     intret = bind (LISTENFD, (structsockaddr*) &address,sizeof(address));  $ASSERT (ret! =-1); Panax Notoginseng  -RET = Listen (LISTENFD,5);  theASSERT (ret! =-1);  +      while(1) { Aprintf"I am worker%d, begin to accept connection.\n", i); the         structsockaddr_in client_addr;  +socklen_t Client_addrlen =sizeof(CLIENT_ADDR);  -         intCONNFD = Accept (LISTENFD, (structsockaddr*) &client_addr, &Client_addrlen);  $         if(CONNFD! =-1) { $printf"worker%d accept a connection success. ip:%s, prot:%d\n", I, Inet_ntoa (client_addr.sin_addr), client_addr.sin_port); -}Else { -printf"worker%d accept a connection failed,error:%s", I, Strerror (errno)); the         } -         CharBuffer[maxline];Wuyi         intNbytes =Read (CONNFD, buffer, MAXLINE); theprintf"read from client is:%s\n", buffer); - Write (connfd, buffer, nbytes); Wu Close (CONNFD); -     } About     return 0; $ } -  - intMain () - { A     inti =0; +      for(i =0; i < WORKER; i++) { theprintf"Create worker%d\n", i); -pid_t PID =fork (); $         /*Child Process*/ the         if(PID = =0) { the worker (i); the         } the         if(PID <0) { -printf"Fork Error"); in         } the     } the     /*Wait Child process*/ About      while(Wait (NULL)! =0) the         ; the     if(errno = =echild) { thefprintf (stderr,"wait error:%s\n", Strerror (errno)); +     } -     return 0; the}

The kernel version of my test machine is:

The test results are as follows:

As you can see from the results, four processes monitor the same IP and port.

4. References

Http://lists.dragonflybsd.org/pipermail/users/2013-July/053632.html

Http://www.blogjava.net/yongboy/archive/2015/02/12/422893.html

Http://m.blog.chinaunix.net/uid-10167808-id-3807060.html

Linux Latest So_reuseport Features