Network programming on Linux Server

The common Linux Server development model is multi-process, multi-threaded and Io multiplexed, that is, select, poll, and Epoll three ways, which are now widely used in the IO model mainly epoll, about the performance of the model compared to select and poll much better, This paper also mainly discusses the model and ignores the other two IO multiplexing models.

Multi-line threads is smaller than multi-process overhead, but it is important to pay attention to the variable mutually exclusive access to ensure thread safety when the main thread passes the data to the child thread.

The Epoll model, introduced in the Linux2.6 kernel, improves some of the obvious design shortcomings in select and has higher efficiency. Mainly reflected in the following aspects:

1. Notification mechanism for Epoll. Unlike the select query for the entire file descriptor array, epoll only cares about "static"-the socket for events such as read-write exceptions to data. But this is not to say that select must be less efficient than epoll, for example, in the case of almost a high level of socket activity, select is very advantageous.

2. Epoll core structure is red black tree + linked list. The red and black trees are used to hold all the monitored file descriptors, and each file descriptor adds the corresponding callback function to the NIC driver. The NIC detects that the corresponding event arrives and calls the callback function to join the linked list, so the list is full of active file descriptors. The red and black trees also guarantee that the file description Fuzan to delete all have a faster speed, such as each time you add a file descriptor before the first to determine whether it is in the red black tree, if there is not repeated insertion.

3. Unlike the memory copy of select, such as Copy_to_user, Copy_from_user mode, Epoll mode, the communication between the kernel and the user is through shared memory, directly to memory read and write is the fastest kind of IPC (Unix Network Programming Volume 2).

4. Epoll the number of file descriptors that are monitored at the same time is almost unlimited, unlike the maximum of 2048 for select.

Today (May 27, 2016) I've been wondering why it's not possible to combine multithreading with epoll and how to combine them. So I went online to check the following, there are two main methods:

1. Create several threads, then create a epoll_wait io wait on each thread, and multiple threads share Epoll a file descriptor.

2. Create a new thread each time the data reads and writes, and the data reads and writes.

Which one is better? The references tell me it's the first.

The second has an obvious problem: each time there is data to re-create a new thread to process the data, the creation of the thread also requires a certain amount of overhead (time and memory), and the introduction of Epoll is to try to eliminate multithreading multi-process brought into the overhead. On the other hand, the process is destroyed after the child threads have been processed, because if the thread is not destroyed, a memory leak will occur, and each thread will need 8G for 8m,1000 threads.

The first approach is perfect, taking advantage of the full resources of the CPU and leveraging the benefits of IO Multiplexing: Multiple file descriptors can be monitored in one thread. This is because multiple threads share the Epoll descriptor, and several epoll functions Epoll_ctl, epoll_wait, and so on are thread-safe, so you can reuse a Epoll descriptor in multiple threads.

Here is a reference code written by a foreigner:

　　

/** Copyright (C) 2012-2014 www.56.com Email:jingchun.zhang at renren-inc.com; Jczhang at 126.com; Danezhang77 at gmail.com* * 56VFS is copied only under the terms of the GNU general public License v3*/#include "vfs_s O.h "#include" vfs_init.h "#include" solib.h "#include" myepoll.h "#include" thread.h "#include" myconfig.h "#include" Fdinfo.h "#include" global.h "#include" mybuff.h "#include" log.h "#include" watchdog.h "#include <dlfcn.h># Include <stdio.h> #include <stdlib.h> #include <sys/prctl.h> #include <sys/sendfile.h> #if __ gnuc__ < 4static inline void barrier (void) {__asm__ volatile (""::: "Memory");} #elsestatic inline void barrier (void) {__sync_synchronize ();} #endifstatic int epfd;static int lfd;static int maxevent;static struct mylib solib;static int sub_init_signalling (char *so {Solib.handle = Dlopen (So, Rtld_now), if (Solib.handle = = NULL) {LOG (GLOGFD, Log_error, ' open%s err%s\n ', So, Dlerror ()); return-1;} Solib.svc_init = (proc_init) dlsym (solib.hAndle, "Svc_init"), if (Solib.svc_init) {if (Solib.svc_init () < 0) {LOG (GLOGFD, Log_error, "Svc_init ERROR%m!\n"); return-1;}} Else{log (GLOGFD, Log_error, "Svc_init must be imp!\n"); return-1;} Solib.svc_pinit = (proc_init) dlsym (Solib.handle, "Svc_pinit"); solib.svc_initconn = (Proc_method) dlsym (Solib.handle, "Svc_initconn"); solib.svc_recv = (Proc_method) dlsym (Solib.handle, "svc_recv"); solib.svc_send = (Proc_method) dlsym ( Solib.handle, "Svc_send"), Solib.svc_finiconn = (Proc_fini) dlsym (Solib.handle, "Svc_finiconn"); solib.svc_timeout = ( Proc_timeout) Dlsym (Solib.handle, "svc_timeout"); solib.svc_send_once = (Proc_method) dlsym (Solib.handle, "Svc_send_ Once "); if (Solib.svc_recv && solib.svc_send) return 0; LOG (GLOGFD, Log_error, "svc_send and Svc_recv must be imp!\n"); return-1;} void add_fd_2_efd (int fd) {fcntl (fd, F_SETFL, o_rdwr| O_nonblock); struct conn *curconn = &AMP;ACON[FD];CURCONN-&GT;FD = Fd;curconn->send_len = 0;memset (curconn-> PeerIP, 0, sizeof (Curconn->peerip)); mybuff_rEinit (& (Curconn->send_buff)); Mybuff_reinit (& (Curconn->recv_buff)); uint32_t ip = Getpeerip (FD); IP2STR (Curconn->peerip, IP); LOG (GLOGFD, Log_debug, "fd [%d] [%s]set OK%d\n", FD, Curconn->peerip, CURCONN-&GT;FD), Epoll_add (EPFD, FD, Epollin);} void modify_fd_event (int fd, int events) {events = Epollin|events;epoll_mod (EPFD, FD, events); LOG (GLOGFD, Log_debug, "fd [%d] be modify!\n", FD);} int get_client_data (int fd, char **data, size_t *len) {struct Conn *curcon = &acon[fd];if (Mybuff_getdata (& ( Curcon->recv_buff), data, Len) Return-1;return 0;} int consume_client_data (int fd, size_t len) {struct Conn *curcon = &acon[fd];mybuff_skipdata (& (curcon->recv_ (buff), Len); return 0;} int set_client_data (int fd, char *buf, size_t len) {struct Conn *curcon = &acon[fd];mybuff_setdata (& (curcon-> Send_buff), buf, Len); return 0;} int set_client_fd (int fd, int localfd, size_t offset, size_t len) {struct Conn *curcon = &acon[fd];mybuff_setfile (& (Curcon->seNd_buff), LOCALFD, offset, Len); return 0;}   static void Accept_new () {struct sockaddr_in addr;socklen_t len; int FD = 0;while (1) {fd = Accept (LFD, (struct sockaddr *) &addr, (len = sizeof (addr), &len)); if (FD < 0) Brea K;if (FD >= Maxfds) {LOG (GLOGFD, Log_error, "FD overflow![ %d] > [%d]\n], FD, MAXFDS); close (FD); continue;} if (Solib.svc_initconn (FD)) {LOG (GLOGFD, Log_error, "FD init err![ %d]%m\n ", FD); close (FD); continue;} ADD_FD_2_EFD (FD);}}  void do_close (int fd) {if (fd >= 0 && FD < Maxfds) {struct Conn *curcon = &acon[fd];if (CURCON-&GT;FD < 0) LOG (GLOGFD, Log_error, "FD%d already be closed%s\n", FD, FUNC); LOG (GLOGFD, Log_debug, "%s:%s:%d close fd%d\n", ID, FUNC, LN, FD), if (solib.svc_finiconn) solib.svc_finiconn (FD); barrier (); Epoll_del (EPFD, fd); curcon->fd = -1;close (FD);}}  static void Do_send (int fd) {LOG (GLOGFD, Log_trace, "%s:%s:%d\n", ID, FUNC, LN); int ret = Send_add_epollin;int n = 0;struct Conn *curcon = &acon[fd];if (CURCON-&GT;FD <0) {LOG (GLOGFD, Log_debug, "FD%d already be closed%s\n", FD, FUNC); return;} int localfd;off_t start;char* data;size_t len;if (!mybuff_getdata (& (Curcon->send_buff), &data, &len)) { LOG (GLOGFD, Log_trace, "fd[%d] get len from data [%d]\n", FD, Len); while (1) {n = Send (fd, data, Len, msg_dontwait | msg_nosignal); if (n > 0) {LOG (GLOGFD, Log_trace, "fd[%d] send len%d, datalen%d\n", FD, N, Len); Mybuff_skipdata (& (c Urcon->send_buff), n); if (n < len) ret = Send_add_epollout;curcon->send_len + = n;} else if (errno = = eintr) Continue;else if (errno = = eagain) ret = Send_add_epollout;else{log (GLOGFD, Log_error, "%s:%s:%d FD [%d] Send err%d:%d:%m\n ", ID, FUNC, LN, FD, N, len); ret = Send_close;} Break;}} if (ret = = Send_add_epollin &&!mybuff_getfile (& (Curcon->send_buff), &AMP;LOCALFD, &start, &len ) {LOG (GLOGFD, Log_trace, "fd[%d] get len from file [%d]\n", FD, Len); size_t len1 = len > gsize? Gsize:len;while (1) {n = sendfile64 (fd, LOCALFD, &start, len1); if (n > 0) {mybuff_skipfile (& (Curcon->send_buff), n); LOG (GLOGFD, Log_trace, "%s:%s:%d fd[%d] Send len%d, datalen%d\n", ID, FUNC, LN, FD, N, len1); if (n < len) ret = send_a Dd_epollout;curcon->send_len + = n;} else if (errno = = eintr) Continue;else if (errno = = eagain) ret = send_add_epollout;else {LOG (GLOGFD, Log_error, "%s:%s:%d F D[%D] Send err%d:%d:%m\n ", ID, FUNC, LN, FD, N, len); ret = Send_close;} Break;}} Switch (ret) {case send_close:do_close (FD); Return;case send_add_epollin:modify_fd_event (FD, Epollin); Break;case SEND _add_epollout:modify_fd_event (FD, epollout); Break;case send_add_epollall:modify_fd_event (FD, EPOLLOUT| Epollin); Break;default:modify_fd_event (FD, Epollin); if (n > 0 && solib.svc_send_once) solib.svc_send_once (FD); if (ret = = Send_add_epollin && solib.svc_ Send) if (solib.svc_send (FD) = = Send_close) {LOG (GLOGFD, Log_debug, "%s:%s:%d send CLOSE%d\n", ID, FUNC, LN, FD);d O_close (f d);}} static void Do_recv (int fd) {struct Conn *curcon= &acon[fd];if (CURCON-&GT;FD < 0) {LOG (GLOGFD, Log_error, "FD%d already be closed%s\n", FD, FUNC); return;} Char iobuf[20480] = {0x0};int n = -1;while (1) {n = recv (fd, Iobuf, sizeof (IOBUF), msg_dontwait), if (n > 0) {LOG (GLOGFD, Log_debug, "fd[%d] recv len%d\n", FD, N), Mybuff_setdata (& (Curcon->recv_buff), Iobuf, N), if (n = = sizeof (IOBUF)) {L OG (GLOGFD, Log_debug, "fd[%d] need recv nextloop%d\n", FD, N); continue;} break;} if (n = = 0) {LOG (GLOGFD, Log_error, "fd[%d] close%s:%d!\n", FD, ID, LN); return Do_close (FD); if (errno = = eintr) {LOG (GLOGFD, Log_trace, "fd[%d] need recv again!\n", FD); continue;} else if (errno = = Eagain) {LOG (GLOGFD, Log_trace, "fd[%d] need recv next!\n", FD); Modify_fd_event (FD, Epollin); break;} Else{log (GLOGFD, Log_error, "fd[%d] close%s:%d!\n", FD, ID, LN); return Do_close (FD);}} if (n <= 0) return;int ret = SOLIB.SVC_RECV (FD); switch (ret) {case Recv_close:log (GLOGFD, Log_error, "fd[%d] CLOSE%s:%d! \ n ", FD, ID, LN);d o_close (FD); Break;case Recv_send:do_senD (FD); Break;case recv_add_epollin:modify_fd_event (FD, Epollin); Break;case recv_add_epollout:modify_fd_event (FD, epollout); Break;case recv_add_epollall:modify_fd_event (FD, epollout| Epollin); Break;default:modify_fd_event (FD, Epollin); break;}} static void do_process (int fd, int events) {if (!) ( Events & (Epollin | Epollout)) {LOG (GLOGFD, Log_debug, "Error event%d,%d\n", events, FD);d o_close (FD); return;} if (events & Epollin) {LOG (GLOGFD, Log_trace, "read event%d,%d\n", events, FD);d o_recv (FD);} if (events & Epollout) {LOG (GLOGFD, Log_trace, "Send event%d,%d\n", events, FD);d o_send (FD);}} static void set_socket_attr (int fd) {setsockopt (fd, Sol_socket, So_sndbuf, &init_buff_size, sizeof (int)); SetSockOpt (FD, Sol_socket, So_rcvbuf, &init_buff_size, sizeof (int));} static void So_thread_entry (void *arg) {Pthread_detach (pthread_self ()); if (solib.svc_pinit) {if (Solib.svc_pinit () < 0) {LOG (GLOGFD, Log_error, "Svc_pinit ERROR%m!\n"); stop = 1;return;}} Else{log (GLOGFD, Log_error, "sVc_pinit must be imp!\n "); stop = 1;return;} int n = 0, i = 0;time_t last = time (NULL), time_t now = last;struct epoll_event *pev = (struct epoll_event*) malloc (sizeof (s Truct epoll_event) * maxevent), if (Pev = = NULL) {LOG (GLOGFD, Log_error, "Allocate epoll_event (%d):%m\n", maxevent); stop = 1;return;} while (!stop) {n = epoll_wait (EPFD, Pev, maxevent, +); for (i = 0; i < n; i++) {if (pev[i].data.fd = LFD) accept_new (); Elsedo_process (PEV[I].DATA.FD, pev[i].events);} now = Time (NULL), if (now > last + g_config.cktimeout) {last = Now;if (solib.svc_timeout) solib.svc_timeout ();}}} int vfs_signalling_thread (void *arg) {T_thread_arg *ARGP = (T_thread_arg *) arg;if (sub_init_signalling (argp->name)) {stop = 1;return-1;} if (Argp->port > 0) {LFD = Get_listen_sock (Argp->port), if (LFD < 0) {LOG (GLOGFD, Log_error, "Get_listen_sock er R%d\n ", argp->port); stop = 1;return-1;} LOG (GLOGFD, Log_debug, "%s Listen on%d\n", Argp->name, Argp->port);}    Maxevent = argp->maxevent; EPFD = EPoll_create (maxevent); if (EPFD < 0) {LOG (GLOGFD, Log_error, "epoll_create (%d):%m\n", maxevent); stop = 1;return-1;} if (Argp->port > 0) {if (Argp->flag) set_socket_attr (LFD);} struct Threadstat *thst = Get_threadstat (); int event = epollin;if (Argp->port > 0) epoll_add (EPFD, LFD, event);   LOG (GLOGFD, Log_debug, "%s:%s:%d\n", ID, FUNC, LN); int i = 0;   pthread_attr_t attr; pthread_t Tid;pthread_attr_init (&attr); int rc;if (argp->threadcount <= 0) Argp->threadcount = 1;for (; I < argp->threadcount; i++) {usleep), if (rc = Pthread_create (&tid, &attr, (void* (*) (void*)) So_thread_entry, NULL))! = 0) {LOG ( GLOGFD, Log_error, "Pthread_create err%m\n"); stop = 1;return-1;}} while (!stop) {thread_reached (THST); sleep (200);} return 0;}

Reference:

Http://stackoverflow.com/questions/7058737/is-epoll-thread-safe

Https://github.com/jingchunzhang/56vfs/blob/master/network/vfs_so_r.c

Http://stackoverflow.com/questions/14584833/multithreaded-epoll

Network programming on Linux Server