Linux epoll Development Guide-[ffrpc source code analysis]

Epoll's question has long been like writing an article to talk about his own views, but as ffrpc has not been completed, it will be dragged down. Epoll is mainly used in Server programming. This article mainly discusses epoll usage skills in server programs. Epoll is generally used in combination with asynchronous io, so this article discusses the following application scenarios:

• This article mainly discusses the use of epoll in server programs, mainly involving APIs related to tcp socket.
• Tcp socket is an asynchronous mode, including asynchronous socket read/write and listener asynchronous operations.
• This article will not discuss the details of the API too much, but focuses on the process and design.

Epoll io Model

Epoll is designed for asynchronous io operations. IO events in epoll are divided into read events and write events. If you are in contact with the linux driver module or linux io model, it is easier to understand. In Linux, I/O operations are abstracted as read, write, close, and ctrl operations. Therefore, epoll only provides read, write, and error events, which are consistent with the linux I/O model.

• When epoll notifies the read event, you can call the io system to call read to read data.
• When epoll notifies the write event, you can call the io system to call write to send data.
• When an error occurs, you can close and Recycle resources.
• Ctrl-related interfaces are used to set non-blocking options for socket.

Why do we need to understand the epoll I/O model? In this article, we believe that in some cases, the complexity of the epoll operation code is caused by the mismatch between the model (or class design) in the Code and the epoll I/O model. In other words, if our encoding model matches the epoll io model, the non-blocking socket encoding will be very simple and clear.

The class relationships constructed based on the epoll model are:

typedef  ~ socket_fd_t socket()          =   handle_epoll_read()  =   handle_epoll_write() =   handle_epoll_del()          =   close()                          =  i = , nfds = = (nfds <  && EINTR ===  (i = ; i < nfds; ++& cur_ev =* fd_ptr            = (fd_i* (cur_ev.data.ptr == ) ( ==  (cur_ev.events & (EPOLLIN |->(cur_ev.events &-> (cur_ev.events & (EPOLLERR |->(nfds >=  

 

Comparison of Epoll's LT mode and ET Mode

First, compare the differences between level trigger and edge trigger modes.

The LT mode has the following features:

• If the data is readable, epoll returns a readable event.
• If developers do not fully read the data, epoll will continuously notify the data to be readable until all the data is read.
• If the socket is writable, epoll returns a writable event, and notifies the writable event as long as the socket sends a buffer not full.
• The advantage is that the read operation is relatively simple, as long as there are read events, read more read less can be.
• The disadvantage is that write-related operations are complicated. Because the socket must not send the buffer in the Idle State, if the socket is always in the epoll wait list, epoll will always notify the write event, therefore, when no data is sent, you must delete the socket write event from the epoll wait list. If necessary, add it back. This is the most complex part of the LT mode.

The characteristics of the ET mode are:

• If the socket is readable, a readable event is returned.
• If the developer does not read all the data, epoll will not notify the epoll read event again. That is to say, there is a hidden danger. If the developer does not read all the data when reading the read event, this may cause epoll not to notify the read event of the socket. (In fact, this problem does not sound difficult. refer to the following section ).
• If the sending buffer is not full, epoll notifies the write event until the developer fills up the sending buffer, and epoll notifies the write event when the next sending buffer is changed from full to full.
• In ET mode, only the status of the socket changes, that is, the read event is notified when the buffer is read from no data to when there is data, and the write event is notified when the buffer is sent from full to under full.
• The disadvantage is that when the epoll read event is triggered, you must ensure that all data in the buffer zone read by the socket is read (in fact, this requirement is easily met)
• Advantage: For write events, a notification is sent when the sending buffer is full to full. If there is no data to write, ignore this event. If there is data to write, write directly. The write event of the Socket can always be sent to the wait list of epoll. In Man epoll, we know that when writing data to a socket, the returned value is smaller than the input buffer size or when the write system calls the response to EWouldBlock, it indicates that the sending buffer is full.

Let's look at the ET mode from another perspective. In fact, epoll's ET mode is actually the socket io full state machine.

First, let's look at the read status chart in epoll:

 

 

When the socket changes from unreadable to readable, The epoll ET mode returns the read event. For read events, developers must ensure that all data in the read buffer is read. man epoll knows:

• The Read system calls EwouldBlock, indicating that all data in the Read buffer is Read.
• The value returned by the Read system call is smaller than the passed buffer parameter, indicating that all Read data in the Read buffer is Read.

Sample Code

 nread = = ::read(m_fd, recv_buffer, (recv_buffer) -  (nread > = ->handle_read( (nread < ((recv_buffer) - ;  ( == nread) -> - (errno ==  (errno ==-> -( 

Let's look at the write state machine:

 

It should be noted that the socket mode is writable, because the initial time and space of the sending buffer zone. Therefore, when data is sent at the application layer, the write System Call is called directly to send data. If the write System Call returns EWouldBlock, the socket becomes unwritable, or the value returned by the write System Call is smaller than the size of the passed buffer parameter. In this case, you need to store the unsent data in the application layer to be sent list and wait for epoll to return the write event, then, send the data in the application-level list to be sent. If the data in the application-level list to be sent is not completely sent at a time, wait until epoll returns the write event. Therefore, the following conclusions can be obtained through reverse push. If there is data in the list to be sent at the application layer, the socket must not be written. In this case, new data must be sent and directly appended to the list to be sent. If the list to be sent is empty, it indicates that the socket is writable. You can directly call the write system to send data. Summary:

• When sending data, if there is data in the application-layer list to be sent, the data to be sent is appended to the list to be sent. Otherwise, the write system is called directly.
• When the Write system calls to send data, it checks the write return value. If the returned value is greater than 0 and smaller than the input buffer parameter size, or the EWouldBlock error code is returned, it indicates that the sending buffer is full, append unsent data to the list to be sent
• After Epoll returns the write event, it checks whether there is data in the list to be sent. If there is data, it tries to send all the guidance data in sequence, or the sending buffer is filled up.

Sample Code:

 socket_impl_t::send_impl( & buff_ = ( == is_open() || m_sc->check_pre_send(     ( == ret = (ret <  ->  (ret >         m_sc->handle_write_completed( ret =  ( == is_open() ||  ==  & msg == (ret <  -> -  (ret >   ( ==->handle_write_completed( 

Summary

The <mode is simple in read operations, but it has no advantage in the ET mode because it is not difficult to read all the data in the read buffer. While the write operation and the ET mode are very clear, it is very easy to understand and implement according to the full state machine. The write operations in the LT mode are much more complicated. You need to frequently maintain the wail list of epoll.

When writing code, use epoll ET as the state machine. When the socket is created (the socket returned by the accept and connect system call), it is added to the epoll list, and you do not need to delete it from it. Why? The FAQ in man epoll tells us that when the socket is closed, it is automatically deleted from epoll. Some notes for the socket listener:

• Ignore the write event of listening socket
• The read event of the listening socket indicates that a new connection exists, and the accept is called to accept the connection until EWouldBlock is returned.
• For Error events, some errors are acceptable, such as errors when file descriptors are used up.

Sample Code:

=  new_fd = - ((new_fd = ::accept(m_listen_fd, ( sockaddr *)&addr, &addrlen)) == - (errno ==   (errno == EINTR || errno == EMFILE || errno == ECONNABORTED || errno == ENFILE ||== EPERM || errno == ENOBUFS || errno ==);                m_epoll->mod_fd(  -* socket =-> ( 

GitHub: https://github.com/fanchy/FFRPC

Ffrpc Introduction: http://www.cnblogs.com/zhiranok/p/ffrpc_summary.html