(OK) Linux Epoll model-socket epoll Server client Chat-pthread

Http://www.cnblogs.com/venow/archive/2012/11/30/2790031.html

http://blog.csdn.net/denkensk/article/details/41978015

Definition:

　　The Epoll is an enhanced version of the Linux kernel for processing large numbers of handles and is a poll of the Linux Select/poll IO interface, which significantly reduces the CPU utilization of the program in cases where there is only a small amount of activity in a large number of concurrent connections. Because it will reuse the file descriptor collection to pass the result instead of forcing the developer to re-prepare the set of file descriptors to be listened to each time it waits for the event, another reason is that it does not have to traverse the entire set of descriptors that are being listened to when the event is fetched. Just walk through the set of descriptors that are joined to the ready queue by a kernel IO event that wakes up asynchronously. Epoll provides edge triggering (edge triggered) in addition to the level triggered of Select\poll IO events, which makes it possible for user space programs to cache IO status and reduce epoll_wait/ The epoll_pwait call, which provides the efficiency of the application.

Working mode:

LT (level triggered): Horizontal trigger, default mode, support both block and no-block socket, in this practice, the kernel tells us whether a file descriptor is ready, if ready, you can do the IO operation of this ready FD. If you do not do anything, the kernel will continue to notify you, so this mode of programming error is less likely. Traditional Select\poll are the representatives of this model.

　　ET (edge-triggered): Edge trigger, high-speed mode of operation, only support no-block socket. In this mode, when the descriptor is never ready to become ready, the kernel tells you through Epoll. It then assumes that you know that the file descriptor is ready and no more ready notifications are sent for that descriptor until you do something that causes the file descriptor to no longer be ready (for example, you are sending, accepting, or accepting requests, Or a ewouldblock error is caused by sending the accepted data less than a certain amount). Note, however, that the kernel does not send more notifications if the FS does not do IO operations (causing it to become not ready again).

　Difference : LT event will not be discarded, but as long as the read buffer inside the data can let the user read, then constantly notify you. ET is only notified when the event occurs.

How to use:

1,int epoll_create (int size)

Create a epoll handle with the parameter size to tell the kernel how many to listen.

2,int epoll_ctl (int epfd, int op, int fd, struct epoll_event *event)

Epoll Event Registration function,

The parameter EPFD is the handle of epoll;

The parameter op represents the action, represented by 3 macros: Epoll_ctl_add (Register the new FD to EPFD), Epoll_ctl_mod (Modify the listener event of the registered FD), Epoll_ctl_del (delete a fd from EPFD);

The parameter fd is the identifier that needs to be monitored;

The parameter event tells the kernel what to listen for, the structure of the event is as follows:

    struct Epoll_event {          __uint32_t events;/* Epoll events */          epoll_data_t data;/* USER data variable */        };

Where events can be set up with the following several macros:

　　Epollin: Indicates that the corresponding file descriptor can be read (including a graceful shutdown of the peer socket)

Epollout: Indicates that the corresponding file descriptor can be written

Epollpri: Indicates that the corresponding file descriptor has an urgent data readable (this should indicate that there is out-of-band data coming)

　　Epollerr: Indicates an error occurred for the corresponding file descriptor

Epollhup: Indicates that the corresponding file descriptor is hung up;

Epollet: Set Epoll to edge triggered mode, which is relative to the horizontal trigger (level triggered)

Epolloneshot: Listen to only one event, when the event is monitored, if you still need to continue to listen to the socket, you need to add the socket to the Epoll queue again

3, int epoll_wait (int epfd, struct epoll_event * events, int maxevents, int timeout)

　　Waits for the event to occur, similar to a select () call. Parameter events is used to get the collection of event from the kernel, the maxevents tells the kernel how big the events are, the value of this maxevents cannot be greater than the size when the Epoll_create () was created, the parameter timeout is the timeout (in milliseconds, 0 returns immediately , 1 will be uncertain, and there are statements that are permanently blocked). The function returns the number of events that need to be processed, such as returning 0 to indicate a timeout.

Application Examples:

Below, I refer to a simple program written by someone else in Google code to illustrate it. SVN path: http://sechat.googlecode.com/svn/trunk/

The program a simple chat room program, written in Linux C + +, the server is mainly implemented with the Epoll model, support high concurrency, I test when there are 10,000 client Connection server, the server processing time of less than 1 seconds, of course, the client is only connected with the server, Accepts the server's welcome message, and does not do any other communication. Although the program is relatively simple, we also provide a way of thinking when we consider the high concurrency of the server. In this program, I have taken all the debugging information and some information unrelated to Epoll, and add the necessary comments, it should be easy to understand.

The program contains a total of 3 header files and 3 CPP files. Of the 3 CPP files, each CPP file is an application, Server.cpp: Server program, Client.cpp: Single client program, Tester.cpp: Simulate high concurrency, turn on 10,000 clients to connect to the server.

The utils.h header file contains a set socket as a non-blocking function, as follows:

    #include <fcntl.h>    int setnonblocking (int sockfd)    {        CHK (fcntl (SOCKFD, F_SETFL, Fcntl (SOCKFD, f_ GETFD, 0) | O_nonblock));        return 0;    }

local.h header files, definitions of some constants and declarations of functions are as follows:

    #include "head.h"    #define BUF_SIZE        //default buffer    #define SERVER_PORT 11111    //Listening Port    #define Server_ HOST "127.0.0.1"    //server IP address    #define EPOLL_RUN_TIMEOUT-1    //epoll Time out    #define EPOLL_SIZE 10000    Epoll the maximum number of clients that are listening    #define Str_welcome "WELCOME to sechat!  You ID is:client #%d "    #define Str_message" Client #%d>>%s "    #define str_noone_connected" Noone CONNECTED To server expect you! "    #define CMD_EXIT "Exit"    //two useful macro definitions: Check and assign and detect    #define CHK (eval) if (eval < 0) {perror ("eval"); EXIT ( -1);}    #define CHK2 (res, eval) if (res = eval) < 0) {perror ("eval"); exit ( -1);}    int setnoblock (int sockfd);    int handle_message (int new_fd);    void* handle_message (void *arg);

head.h file, the header file to be used in the program, as follows:

    #ifndef __head__h__    //#define __HEAD__H__    #include <stdio.h>    #include <stdlib.h>    # Include <unistd.h>    #include <string.h>    #include <time.h>    #include <netinet/in.h >    #include <arpa/inet.h>    #include <sys/socket.h>    #include <sys/epoll.h>    # Include <pthread.h>    #include <iostream>    #include <list>    #include <iterator>    //#endif

Server.cpp file, the Epoll model is implemented here, as follows:

    #include "local.h" #include "utils.h" #include "head.h" using namespace std;    List List < int >clients_list; that holds the client socket descriptor        int main (int argc, char *argv[]) {int listener;        Monitor socket struct sockaddr_in addr, their_addr;        addr.sin_family = pf_inet;        Addr.sin_port = htons (Server_port);        ADDR.SIN_ADDR.S_ADDR = inet_addr (server_host);        Socklen_t Socklen;        Socklen = sizeof (struct sockaddr_in);        static struct epoll_event ev, events[epoll_size]; ev.events = Epollin |    Epollet;        Interested in reading, Edge triggering char message[buf_size];        int EPFD;        Epoll descriptor clock_t Tstart;        Calculate program run time int client, res, epoll_events_count;    CHK2 (Listener, Socket (pf_inet, sock_stream, 0));    Initialize the listener socket setnonblocking (listener);        Set the listener socket to not block//SET socket option to avoid address using error int on=1;  if ((SetSockOpt (listener,sol_socket,so_reuseaddr,&on,sizeof (ON))) <0) {          Perror ("setsockopt failed");        Exit (Exit_failure);    } CHK (Bind (listener, (struct sockaddr *) &addr, sizeof (addr)));    Bind monitor Socket CHK (Listen (listener, 1));    Set monitoring CHK2 (EPFD, Epoll_create (epoll_size));        Create a Epoll descriptor and add the listener socket to epoll EV.DATA.FD = listener;        CHK (Epoll_ctl (EPFD, Epoll_ctl_add, Listener, &ev));            while (1) {CHK2 (Epoll_events_count, epoll_wait (EPFD, events, Epoll_size, epoll_run_timeout));            Tstart = Clock (); for (int i = 0; i < Epoll_events_count; i++) {if (EVENTS[I].DATA.FD = = listener)//New connection arrives, add connection to Epol L, and send the Welcome message {CHK2 (client, accept (listener, struct sockaddr *) &their_addr, &sock                    Len));                    setnonblocking (client);                    EV.DATA.FD = client;                    CHK (Epoll_ctl (EPFD, Epoll_ctl_add, client, &ev));    Clients_list.push_back (client); // Add a new client to list bzero (message, buf_size);                    res = sprintf (message, str_welcome, client);                    CHK2 (res, send (client, message, Buf_size, 0));                printf ("Client:%d\n", client);    } else {//chk2 (res, handle_message (EVENTS[I].DATA.FD)); Note: This does not call Epoll_ctl to reset the socket's event type, but still can continue to receive the message sent by the client//printf ("-------------client:%d\n", events[i                    ].DATA.FD);                    * pthread_t reader;                    int arg = EVENTS[I].DATA.FD;                    int rt = Pthread_create (&reader, NULL, Handle_message, (void *) &arg);                        if ( -1 = = RT) {printf ("Thread creation error\n");                    return-1; }//*/}} printf ("Statistics:%d events handled at:%.2f second (s \ n ", Epoll_events_count, (double) (Clock ()-Tstart)/CLOCKS_PER_SEC);            /* statistics:1 Events handled at:0.00 second (s) ... Statistics:1 Events handled at:0.00 second (s) statistics:20 events handled at:0.01 second (s) Sta tistics:291 Events handled at:0.03 second (s) statistics:2532 events handled at:0.23 second (s) St        atistics:7156 Events handled at:0.31 second (s)//*/} close (listener);        Close (EPFD);        printf ("Is it over\n");    return 0;        }//int handle_message (int client) void* handle_message (void *arg) {int client = * ((int*) arg);        Sleep (5);        Char Buf[buf_size], message[buf_size];        Bzero (buf, buf_size);        Bzero (message, buf_size);        int Len;    CHK2 (Len, recv (client, buf, buf_size, 0));            Accept client Information if (len = = 0)//client off or error, close socket, and remove socket {CHK (Close client) from list;        Clients_list.remove (client); } else           Send message to Client {if (clients_list.size () = = 1) {CHK (Send (client, Str_noone_connec                TED, strlen (str_noone_connected), 0));                return Len;            return NULL;            } sprintf (Message, str_message, client, buf);            List < int >::iterator it; for (it = Clients_list.begin (); It! = Clients_list.end (); it++) {if (*it! = client) {C                HK (Send (*it, message, buf_size, 0));        }}}//return len;    return NULL; }

Tester.cpp file, simulate the high concurrency of the server, open 10,000 clients to connect to the server, as follows:

    #include "local.h" #include "utils.h" #include "head.h" #define Epoll_test_size 8000 using namespace std;        Char Message[buf_size];    Accept Server Information List < int >list_of_clients;    Store all client int res;    clock_t Tstart;        int main (int argc, char *argv[]) {int sock;        struct sockaddr_in addr;        addr.sin_family = pf_inet;        Addr.sin_port = htons (Server_port);        ADDR.SIN_ADDR.S_ADDR = inet_addr (server_host);        Tstart = Clock ();            for (int i = 0; i < epoll_test_size; i++)//Generate Epoll_test_size client, here is 10,000, analog high concurrency {usleep (1000);            CHK2 (sock, socket (pf_inet, sock_stream, 0));            CHK (Connect (sock, (struct sockaddr *) &addr, sizeof (addr)) < 0);            List_of_clients.push_back (sock);            Bzero (message, buf_size);            CHK2 (Res, recv (sock, message, buf_size, 0));            printf ("%s\n", message);        CHK2 (res, send (sock, message, buf_size, 0)); }        List < int >::iterator it;        Remove all client for (it = List_of_clients.begin (); It! = List_of_clients.end (); it++) Close (*it);        printf ("Test passed at:%.2f second (s) \ n", (double) (Clock ()-Tstart)/clocks_per_sec);        printf ("Total server connections is:%d\n", epoll_test_size);    return 0; }

I will not give the results of the implementation of the program, but the following is the code author himself test, you can see, concurrency 10000 no pressure AH

A single client goes to connect to the server, client.cpp file, as follows:

    #include "local.h" #include "utils.h" #include "head.h" using namespace std;    Char Message[buf_size]; /* Process: Call fork to generate two processes, two processes to communicate through the pipeline sub-process: Waiting for customer input, and the information entered by the customer through the pipeline to the parent process parent process: Accept the server information and display, will be from the child        The information received by the process is sent to the server */int main (int argc, char *argv[]) {int sock, PID, pipe_fd[2], EPFD;        struct sockaddr_in addr;        addr.sin_family = pf_inet;        Addr.sin_port = htons (Server_port);        ADDR.SIN_ADDR.S_ADDR = inet_addr (server_host);        static struct epoll_event ev, events[2]; ev.events = Epollin |        Epollet;        Exit flag int continue_to_work = 1;        CHK2 (sock, socket (pf_inet, sock_stream, 0));        CHK (Connect (sock, (struct sockaddr *) &addr, sizeof (addr)) < 0);        CHK (pipe (PIPE_FD));        CHK2 (EPFD, Epoll_create (epoll_size));        EV.DATA.FD = sock;        CHK (Epoll_ctl (EPFD, Epoll_ctl_add, Sock, &ev));        EV.DATA.FD = pipe_fd[0]; CHK (Epoll_ctl (EPFD, Epoll_ctl_add, Pipe_fd[0], &ev));        Call fork to generate two processes CHK2 (PID, fork ());    Switch (PID) {case 0://Sub-process close (pipe_fd[0]);            Turn off read-only printf ("Enter ' exit ' to exit\n");                while (continue_to_work) {bzero (&message, buf_size);                Fgets (Message, buf_size, stdin); When you receive the Exit command, exit if (strncasecmp (message, Cmd_exit, strlen (cmd_exit)) = = 0) {continue_t                o_work = 0;                } else {CHK (write (pipe_fd[1], message, strlen (message)-1));        }} break;    Default://Parent process close (pipe_fd[1]);            Close write-end int epoll_events_count, RES;                while (continue_to_work) {CHK2 (Epoll_events_count, epoll_wait (EPFD, events, 2, epoll_run_timeout));            for (int i = 0; i < Epoll_events_count; i++) {bzero (&message, buf_size);        if (events[i].data.fd = = sock)//accept information from server {CHK2 (res, recv (sock, mess                        Age, Buf_size, 0));                            if (res = = 0)//server closed {CHK (Close (sock));                        continue_to_work = 0;                        } else {printf ("%s\n", message); }} else//Receive information from child process {CHK2 (res, read (EVENTS[I].DATA.FD, ME                        Ssage, buf_size));                        if (res = = 0) {continue_to_work = 0;                        } else {CHK (send (sock, message, buf_size, 0));            }}}}} if (PID) {close (pipe_fd[0]);        Close (sock);        } else {close (pipe_fd[1]);    } return 0; }

Source code Download:
linux-epoll-server-client-chat-pthread.tar.gz

[[email protected] test]# g++ server.cpp-o Server
[Email protected] test]# g++ tester.cpp-o Tester
[Email protected] test]# g++ client.cpp-o Client


Note:

Linux under the input command ulimit-a, you can see open files general 1024, so unmodified when doing this experiment, the largest open to 1023th. So to modify.

Modification method: Ulimit-n < number of files that can be opened at the same time > set the maximum number of files that users can open at the same time

If this parameter is set too small, the too many open files error may occur for sites with large concurrent accesses.

(OK) Linux Epoll model-socket epoll Server client Chat-pthread