"Go" Building a c1000k Server (1) – Basic

Source from Ideawu Building c1000k Server (1) – Basic

When the famous c10k question was put forward, it was 2001, now 12 years later in 2013, c10k is no longer a problem, any ordinary programmer, can use the language and library at hand, easy to write c10k server. This benefits from software advancements as well as improved hardware performance.

Now, it's time to consider c1000k, the problem of millions of connections. Websites like Twitter, Weibo, and Facebook, which have tens of thousands of online users and want messages to be pushed to users in near real time, require the server to maintain a TCP network connection with tens of millions of users, although hundreds of servers can be used to support so many users, but If each server can support 1 million connections (c1000k), then only 10 servers are required.

There are a number of technologies that claim to solve c1000k problems, such as Erlang, Java NIO, and so on, but we should first figure out what is limiting the solution to the c1000k problem. These are the main points:

1. Can the operating system support millions of connections?
2. How much memory does the operating system need to maintain millions of connections?
3. How much memory does an application need to maintain millions of connections?
4. Does the throughput of millions of connections exceed the network limit?

The following is a separate analysis of these issues.

1. Can the operating system support millions of connections?

For the majority of Linux operating systems, c1000k! is not supported by default Because the operating system contains the maximum number of open files (max Open file) restrictions, it is divided into system-wide, and process-level restrictions.

Global limits

Execute under Linux:

Cat/proc/sys/fs/file-nr

Prints a line of output similar to the following:

51000101747

The third number 101747 is the maximum number of open files for the current system (max Open file), and you can see that there is only 100,000, so c1000k cannot be supported on this server. Many systems have a smaller value, in order to modify this value, use the root permission to modify the/E tc/sysctl.conf file:

Fs.file-max = 1020000net.ipv4.ip_conntrack_max = 1020000net.ipv4.netfilter.ip_conntrack_max = 1020000

Process limits

Perform:

Ulimit-n

Output:

1024

Indicates that the current Linux system can only open up to 1024 files per process. To support c1000k, you also need to modify this restriction.

Temporary modification

Ulimit-n 1020000

However, if you are not root, you may not be able to modify more than 1024, error:

-bash:ulimit:open Files:cannot Modify limit:operation not permitted

Permanently modified

To edit the/etc/security/limits.conf file, add the following line:

#/etc/security/limits.confwork         hard    nofile      1020000work         soft    nofile      1020000

The first column work represents the work user, you can fill * in, or root . Then save the exit and log back in to the server.

Note: The Linux kernel source has a constant (Nr_open in/usr/include/linux/fs.h), limiting the maximum number of open files, such as RHEL 5 is 1048576 (2^20), so to support c1000k, you may also need to re-edit Translation kernel.

2. How much memory does the operating system need to maintain millions of connections?

Solve the operating system parameters, the next step is to look at the memory usage. First, the memory consumption of these connections is maintained by the operating system itself. For the Linux operating system, the socket (FD) is an integer, so it is assumed that the operating system manages 1 million connections that occupy memory should be 4m/8m, and then include some management information, should be about 100M. However, there is no analysis of the memory occupied by the socket send and receive buffers. To do this, I wrote the most primitive C-network program to verify:

Server

#include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include < errno.h> #include <arpa/inet.h> #include <netinet/tcp.h> #include <sys/select.h> #define Max_    PORTS 10int Main (int argc, char **argv) {struct sockaddr_in addr;    const char *IP = "0.0.0.0";    int opt = 1;    int bufsize;    Socklen_t Optlen;    int connections = 0;    int base_port = 7000;    if (argc > 2) {base_port = Atoi (argv[1]);    } int server_socks[max_ports];        for (int i=0; i<max_ports; i++) {int port = base_port + i;        Bzero (&addr, sizeof (addr));        addr.sin_family = af_inet;        Addr.sin_port = htons ((short) port);        Inet_pton (Af_inet, IP, &addr.sin_addr);        int serv_sock;        if (Serv_sock = socket (af_inet, sock_stream, 0)) = =-1) {goto sock_err;        } if (setsockopt (Serv_sock, Sol_socket, so_reuseaddr, &opt, sizeof (opt)) = =-1) {goto sock_err;   }     if (Bind (Serv_sock, (struct sockaddr *) &addr, sizeof (addr)) = =-1) {goto sock_err;        } if (Listen (Serv_sock, 1024x768) = =-1) {goto sock_err;        } Server_socks[i] = Serv_sock;    printf ("Server Listen on port:%d\n", port);    }//optlen = sizeof (bufsize);    GetSockOpt (Serv_sock, Sol_socket, So_rcvbuf, &bufsize, &optlen);    printf ("Default send/recv buf Size:%d\n", bufsize);        while (1) {Fd_set readset;        Fd_zero (&readset);        int maxfd = 0;            for (int i=0; i<max_ports; i++) {fd_set (server_socks[i], &readset);            if (Server_socks[i] > maxfd) {maxfd = server_socks[i];        }} int ret = SELECT (Maxfd + 1, &readset, NULL, NULL, NULL);            if (Ret < 0) {if (errno = = eintr) {continue; }else{printf ("Select error!                %s\n ", Strerror (errno));            Exit (0);    }    } if (Ret > 0) {for (int i=0; i<max_ports; i++) {if (!                Fd_isset (Server_socks[i], &readset)) {continue;                } socklen_t Addrlen = sizeof (addr);                int sock = accept (Server_socks[i], (struct sockaddr *) &addr, &addrlen);                if (sock = =-1) {goto sock_err;                } connections + +;            printf ("Connections:%d, FD:%d\n", connections, sock);    }}} return 0;sock_err:printf ("Error:%s\n", Strerror (errno)); return 0;}

Note that the server listens on 10 ports, which is for testing convenience. Because there is only one client test machine that can create more than 30,000 connections to the same IP port, the server listens on 10 ports so that a single test machine can create 300,000 connections to the server.

Client

#include <stdio.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include < errno.h> #include <arpa/inet.h> #include <netinet/tcp.h>int main (int argc, char **argv) {if (argc <= 2)        {printf ("Usage:%s IP port\n", argv[0]);    Exit (0);    } struct sockaddr_in addr;    const char *IP = argv[1];    int base_port = atoi (argv[2]);    int opt = 1;    int bufsize;    Socklen_t Optlen;    int connections = 0;    Bzero (&addr, sizeof (addr));    addr.sin_family = af_inet;    Inet_pton (Af_inet, IP, &addr.sin_addr);    Char tmp_data[10];    int index = 0;        while (1) {if (++index >=) {index = 0;        } int port = base_port + index;        printf ("Connect to%s:%d\n", IP, Port);        Addr.sin_port = htons ((short) port);        int sock;        if (sock = socket (af_inet, sock_stream, 0)) = =-1) {goto sock_err; if (Connect (sock, struct sockaddr *) &addr, sizeof (addr)) = =-1) {goto sock_err;        } connections + +;        printf ("Connections:%d, FD:%d\n", connections, sock);            If (connections% 10000 = = 9999) {printf ("Press Enter to continue:");        GetChar ();        } usleep (1 * 1000);           /* bufsize = 5000;           SetSockOpt (Serv_sock, Sol_socket, So_sndbuf, &bufsize, sizeof (bufsize));         SetSockOpt (Serv_sock, Sol_socket, So_rcvbuf, &bufsize, sizeof (bufsize));    */} return 0;sock_err:printf ("Error:%s\n", Strerror (errno)); return 0;}

I tested 100,000 connections, these connections are idle, and what data is not sent or received. At this point, the process takes up less than 1MB of memory. However, by comparing the free command before and after the program exits, it is found that the operating system uses 200M (approximate) memory to maintain the 100,000 connections! If it is a million connection, the operating system itself will occupy 2GB of memory! That's 2KB per connection.

can modify

/proc/sys/net/ipv4/tcp_wmem/proc/sys/net/ipv4/tcp_rmem

To control the size of the send and receive buffers for TCP connections (thanks @egmkang).

3. How much memory does the application need to maintain millions of connections?

Through the test code above, it can be found that the application maintains millions of idle connections, only consumes the operating system's memory, through the PS command to see that the application itself almost does not occupy memory.

4. Does the throughput of millions of connections exceed the network limit?

Assuming that 20% of millions of connections are active and each connection transmits 1KB of data per second, the required network bandwidth is 0.2M x 1kb/s x 8 = 1.6Gbps, requiring the server to be at least a gigabit network card (10Gbps).

Summarize

Linux systems need to modify kernel parameters and system configuration to support c1000k. C1000K's application requires that the server require at least 2GB of memory, which should be at least 10GB if the application itself requires memory. At the same time, the NIC should be at least a gigabit NIC.

Of course, this is only theoretical analysis, and the actual application requires more memory and CPU resources to process the business data.

Reference:

* http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
* http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/

"Go" Building a c1000k Server (1) – Basic