Muduo network library source code analysis (1) EventLoop event loop (Poller and Channel)

Muduo network library source code analysis (1) EventLoop event loop (Poller and Channel)

Starting from this blog post, we began to analyze the source code of the Muduo network library, mainly combining the "Linux multi-thread Server programming" and some learning materials on the Internet!

(1) Nature of TCP network programming: Three and a half events

1. Establish a connection, including accepting (accept) new connections from the server and successfully initiating (connect) connections from the client. Once a TCP connection is established, the client and server are equal and can send and receive data separately.

2. disconnection, including active disconnection (close or shutdown) and passive disconnection (read (2) returns 0 ).

3. The message arrives and the file descriptor is readable. This is the most important event. The processing method determines the network programming style (blocking or non-blocking, how to handle subcontracting, and how to design the buffer at the application layer ).

3.5 after the message is sent, this is half a message. For low-traffic services, you don't have to worry about this event. In addition, "sent" refers to writing data into the buffer zone of the operating system, the TCP stack is responsible for data transmission and retransmission, which does not mean that the other party has received the data.

Among them, the most important thing is the third point: Message arrival, the file descriptor is readable. Next we will analyze it carefully (by the way, the message sending is analyzed ):

 

(1) Message arrival, file readable:

Kernel reception-> network library readable event trigger --> transfers data from the kernel to the application buffer (and the callback function OnMessage checks whether the data packet is complete based on the protocol, if not immediately returned) --> read, unpack, process, and send (read decode compute encode write)

(2) message sending is completed:

Application Buffer --> kernel buffer (can be fully filled) ---> triggers the event of sending completion and calls back Onwrite. If the kernel buffer is insufficient to accommodate data (high-traffic services), you need to append the data to the kernel sending buffer in the application layer, and then trigger the socket write event. The application layer --> kernel; when all data is sent to the kernel, Onwrite is called back (you can continue to write)

(2) event cycle diagram

EventLoop class:

 

EventLoop is the encapsulation of the Reactor mode. Because Muduo's concurrency prototype is Multiple reactors + threadpool (one loop per thread + threadpool), each thread can have only one EventLoop object. When constructing an EventLoop object, the system checks whether other EventLoop objects have been created in the current thread. If other EventLoop objects have been created, terminate the program (LOG_FATAL ), the EventLoop constructor records the thread (threadld _) to which the object belongs. The thread that creates the EventLoop object is called the IO thread. Its function is to run the event loop (EventLoop: loop ), nothing else =

The following is a simplified EventLoop (the internal Poller is not implemented yet, but it is just a framework)

EventLoop. h

 

# Ifndef MUDUO_NET_EVENTLOOP_H # define MUDUO_NET_EVENTLOOP_H # include
 
  
# Include
  
   
# Include
   
    
Namespace muduo {namespace net {// Reactor, at most one per thread. /// This is an interface class, so don't expose too much details. class EventLoop: boost: noncopyable {public: EventLoop ();~ EventLoop (); // force out-line dtor, for scoped_ptr members. /// Loops forever. /// Must be called in the same thread as creation of the object. void loop (); void assertInLoopThread () {if (! IsInLoopThread () {abortNotInLoopThread () ;}} bool isInLoopThread () const {return threadId _ == CurrentThread: tid ();} static EventLoop * getEventLoopOfCurrentThread (); private: void abortNotInLoopThread (); bool looping _;/* atomic */const pid_t threadId _; // The thread ID of the current object}; }}# endif // MUDUO_NET_EVENTLOOP_H
   
  
 

 

EventLoop. c

 

# Include
 
  
# Include
  
   
# Include
   
    
Using namespace muduo; using namespace muduo: net; namespace {// EventLoop Object Pointer of the current thread // local storage of the thread _ thread EventLoop * t_loopInThisThread = 0;} EventLoop * EventLoop :: getEventLoopOfCurrentThread () {return t_loopInThisThread;} EventLoop: EventLoop (): looping _ (false), threadId _ (CurrentThread: tid ()) {LOG_TRACE <"EventLoop created" <this <"in thread" <threadId _; // if the current thread has already created an EventLoop object, terminate (LOG_FATAL) I F (t_loopInThisThread) {LOG_FATAL <"Another EventLoop" <t_loopInThisThread <"exists in this thread" <threadId _;} else {t_loopInThisThread = this;} EventLoop :: ~ EventLoop () {t_loopInThisThread = NULL;} // event loop. This function cannot be called across threads. // you can only call void EventLoop: loop () in the thread that creates the object () {assert (! Looping _); // assertInLoopThread (); looping _ = true; LOG_TRACE <"EventLoop" <this <"start looping ";: poll (NULL, 0, 5*1000); LOG_TRACE <"EventLoop" <this <"stop looping"; looping _ = false;} void EventLoop :: abortNotInLoopThread () {LOG_FATAL <"EventLoop: abortNotInLoopThread-EventLoop" <this <"was created in threadId _ =" <threadId _ <", current thread id = "<CurrentThread: tid ();}
   
  
 

Poller class:

 

Sequence diagram:

 

 

Poller is an abstract class, which can be EPollPoller (default) or PollPoller. It must be implemented (the only class that uses object-oriented)

For PollPoller, there is a map used to associate fd and channel. We can quickly find the corresponding channel based on fd. One fd corresponds to one struct pollfd (pollfd. fd) and one fd corresponds to one channel *. This fd can be socket, eventfd, timerfd, and signalfd.

Poller is used to update the channel (IO event) in IO reuse and add or delete the Channel. Let's take a look at the implementation of PollPoller:

PollPoller. h

 

# Ifndef MUDUO_NET_POLLER_POLLPOLLER_H # define MUDUO_NET_POLLER_POLLPOLLER_H # include
 
  
# Include
  # Include
   
    
Struct pollfd; namespace muduo {namespace net {// IO Multiplexing with poll (2). class PollPoller: public Poller {public: PollPoller (EventLoop * loop); virtual ~ PollPoller (); virtual Timestamp poll (int timeoutMs, ChannelList * activeChannels); virtual void updateChannel (Channel * channel); virtual void removeChannel (Channel * channel); private: void fillActiveChannels (int numEvents, ChannelList * activeChannels) const; typedef std: vector
    
     
PollFdList; typedef std: map
     
      
ChannelMap; // The key is the file descriptor and the value is Channel * PollFdList pollfds _; ChannelMap channels _ ;}}# endif/MUDUO_NET_POLLER_POLLPOLLER_H
     
    
   
 

 

PollPoller. c

 

# Include
 
  
# Include
  
   
# Include
   
    
# Include
    
     
# Include
     
      
Using namespace muduo; using namespace muduo: net; PollPoller: PollPoller (EventLoop * loop): Poller (loop) {} PollPoller ::~ PollPoller () {} Timestamp PollPoller: poll (int timeoutMs, ChannelList * activeChannels) {// XXX pollfds _ shouldn't change int numEvents =: poll (& * pollfds _. begin (), pollfds _. size (), timeoutMs); Timestamp now (Timestamp: now (); if (numEvents> 0) {LOG_TRACE <numEvents <"events happended"; fillActiveChannels (numEvents, activeChannels);} else if (numEvents = 0) {LOG_TRACE <"nothing happended" ;} Else {LOG_SYSERR <"PollPoller: poll ()";} return now;} void PollPoller: equals (int numEvents, ChannelList * activeChannels) const {for (PollFdList:: const_iterator pfd = pollfds _. begin (); pfd! = Pollfds _. end () & numEvents> 0; ++ pfd) {if (pfd-> revents> 0) {-- numEvents; ChannelMap: const_iterator ch = channels _. find (pfd-> fd); assert (ch! = Channels _. end (); Channel * channel = ch-> second; assert (channel-> fd () = pfd-> fd ); channel-> set_revents (pfd-> revents); // pfd-> revents = 0; activeChannels-> push_back (channel); }}} void PollPoller :: updateChannel (Channel * channel) {Poller: assertInLoopThread (); LOG_TRACE <"fd =" <channel-> fd () <"events =" <channel-> events (); if (channel-> index () <0) {// index <0 indicates a new channel // a new one, add to pollfds _ assert (channels _. find (channel-> fd () = channels _. end (); struct pollfd pfd; pfd. fd = channel-> fd (); pfd. events = static_cast
      
        (Channel-> events (); pfd. revents = 0; pollfds _. push_back (pfd); int idx = static_cast
       
         (Pollfds _. size ()-1; channel-> set_index (idx); channels _ [pfd. fd] = channel;} else {// update existing one assert (channels _. find (channel-> fd ())! = Channels _. end (); assert (channels _ [channel-> fd ()] = channel); int idx = channel-> index (); assert (0 <= idx & idx <static_cast
        
          (Pollfds _. size (); struct pollfd & pfd = pollfds _ [idx]; assert (pfd. fd = channel-> fd () | pfd. fd =-channel-> fd ()-1); pfd. events = static_cast
         
           (Channel-> events (); pfd. revents = 0; // temporarily change a channel to event-free, but not remove this channel from Poller if (channel-> isNoneEvent ()) {// ignore this pollfd // temporarily ignore the file descriptor event // here pfd. fd can be directly set to-1 pfd. fd =-channel-> fd ()-1; // This is set to removeChannel optimization} void PollPoller: removeChannel (Channel * channel) {Poller :: assertInLoopThread (); LOG_TRACE <"fd =" <channel-> fd (); assert (channels _. find (channel-> fd ())! = Channels _. end (); assert (channels _ [channel-> fd ()] = channel); assert (channel-> isNoneEvent ()); int idx = channel-> index (); assert (0 <= idx & idx <static_cast
          
            (Pollfds _. size (); const struct pollfd & pfd = pollfds _ [idx]; (void) pfd; assert (pfd. fd =-channel-> fd ()-1 & pfd. events = channel-> events (); size_t n = channels _. erase (channel-> fd (); assert (n = 1); (void) n; if (implicit_cast
           
             (Idx) = pollfds _. size ()-1) {pollfds _. pop_back ();} else {// the complexity of the removed algorithm is O (1). the elements to be deleted are exchanged with the last element and then pop_back int channelAtEnd = pollfds _. back (). fd; iter_swap (pollfds _. begin () + idx, pollfds _. end ()-1); if (channelAtEnd <0) {channelAtEnd =-channelAtEnd-1;} channels _ [channelAtEnd]-> set_index (idx); pollfds _. pop_back ();}}
           
          
         
        
       
      
     
    
   
  
 

Several tips in the Code are marked in comments.

 

Channel class:

 

A Channel is a selectable IO channel that registers and responds to IO events. It does not have a file descriptor.

 

A Channel is an "Event" in the Reactor structure. It belongs to an EventLoop from start to end (an EventLoop corresponds to multiple channels and processes multiple IO events) and is responsible for the IO events of a file descriptor, it contains and file descriptor fd _, but in fact it does not own fd _ and does not need to be closed. Stores the IO event type and the corresponding callback function in the Channel class. When an IO event occurs, the callback function in the Channel class is called. The Channel class is generally not used independently. It is often included in other classes (Acceptor, Connector, EventLoop, TimerQueue, and TcpConnection. The Channel class has the EventLoop pointer loop _. This pointer can be used to add the current Channel event to EventLoop. The event type is represented by events _. Different event types correspond to different callback functions.

 

Both of the following are registered by the Channel:

Acceptor is the abstraction of passive connections ---> focus on listening to readable events of sockets and callback handleRead.

The abstraction of active connections by ctor.

Sequence diagram:

Channel. h

 

# Ifndef MUDUO_NET_CHANNEL_H # define MUDUO_NET_CHANNEL_H # include
 
  
# Include
  
   
# Include
   
    
# Include
    
     
# Include
     
      
Namespace muduo {namespace net {class EventLoop; // A selectable I/O channel. /// This class doesn' t own the file descriptor. /// The file descriptor cocould be a socket, // an eventfd, a timerfd, or a signalfdclass Channel: boost: noncopyable {public: typedef boost: function
      
        EventCallback; typedef boost: function
       
         ReadEventCallback; Channel (EventLoop * loop, int fd );~ Channel (); void handleEvent (Timestamp receiveTime); void setReadCallback (const ReadEventCallback & cb) {readCallback _ = cb;} void setWriteCallback (const EventCallback & cb) {writeCallback _ = cb;} void setCloseCallback (const EventCallback & cb) {closeCallback _ = cb;} void setErrorCallback (const EventCallback & cb) {errorCallback _ = cb ;} /// Tie this channel to the owner object managed by shared_ptr, // prevent the owner object being destroyed in handleEvent. void tie (const boost: shared_ptr
        
          &); Int fd () const {return fd _;} int events () const {return events _;} void set_revents (int revt) {revents _ = revt ;} // used by pollers // int revents () const {return revents _;} bool isNoneEvent () const {return events _ = kNoneEvent;} void enableReading () {events _ | = kReadEvent; update () ;}// void disableReading () {events _ & = ~ KReadEvent; update () ;}void enableWriting () {events _ | = kWriteEvent; update () ;} void disableWriting () {events _ & = ~ KWriteEvent; update ();} void disableAll () {events _ = kNoneEvent; update ();} bool isWriting () const {return events _ & kWriteEvent ;} // for Poller int index () {return index _;} void set_index (int idx) {index _ = idx;} // for debug string reventsToString () const; void doNotLogHup () {logHup _ = false;} EventLoop * ownerLoop () {return loop _;} void remove (); private: void update (); void handleEventWithGuard (Timestamp receiveTime); static const int kNoneEvent; static const int kReadEvent; static const int kWriteEvent; EventLoop * loop _; // EventLoop const int fd _; // file descriptor, but is not responsible for disabling the int events _; // The Event int revents _; // poll/epoll returned event int index _; // used by Poller. indicates the serial number bool logHup _ in the poll event array; // for POLLHUP boost: weak_ptr
         
           Tie _; bool tied _; bool eventHandling _; // whether the event is being processed ReadEventCallback readCallback _; EventCallback writeCallback _; EventCallback closeCallback _; EventCallback errorCallback _;};}} # endif // MUDUO_NET_CHANNEL_H
         
        
       
      
     
    
   
  
 

Channel. c

 

 

# Include
 
  
# Include
  
   
# Include
   
    
# Include
    
     
# Include
     
      
Using namespace muduo; using namespace muduo: net; const int Channel: kNoneEvent = 0; const int Channel: kReadEvent = POLLIN | POLLPRI; const int Channel: kWriteEvent = POLLOUT; channel: Channel (EventLoop * loop, int fd _): loop _ (loop), fd _ (fd _), events _ (0 ), revents _ (0), index _ (-1), logHup _ (true), tied _ (false), eventHandling _ (false) {} Channel ::~ Channel () {assert (! EventHandling _);} void Channel: tie (const boost: shared_ptr
      
        & Obj) {tie _ = obj; tied _ = true;} void Channel: update () {loop _-> updateChannel (this );} // before calling this function, make sure to call disableAllvoid Channel: remove () {assert (isNoneEvent (); loop _-> removeChannel (this);} void Channel :: handleEvent (Timestamp receiveTime) {boost: shared_ptr
       
         Guard; if (tied _) {guard = tie _. lock (); if (guard) {handleEventWithGuard (receiveTime) ;}} else {receive (receiveTime) ;}} void Channel: handleEventWithGuard (Timestamp receiveTime) {eventHandling _ = true; if (revents _ & POLLHUP )&&! (Revents _ & POLLIN) {if (logHup _) {LOG_WARN <"Channel: handle_event () POLLHUP";} if (closeCallback _) closeCallback _();} if (revents _ & POLLNVAL) {LOG_WARN <"Channel: handle_event () POLLNVAL";} if (revents _ & (POLLERR | POLLNVAL) {if (errorCallback _) errorCallback _ ();} if (revents _ & (POLLIN | POLLPRI | POLLRDHUP) {if (readCallback _) readCallback _ (receiveTime);} if (revents _ & POLLOUT) {if (writeCallback _) writeCallback _ ();} eventHandling _ = false;} string Channel: reventsToString () const {std: ostringstream oss; oss <fd _ <":"; if (revents _ & POLLIN) oss <"IN"; if (revents _ & POLLPRI) oss <"PRI "; if (revents _ & POLLOUT) oss <"OUT"; if (revents _ & POLLHUP) oss <"HUP"; if (revents _ & POLLRDHUP) oss <"RDHUP"; if (revents _ & POLLERR) oss <"ERR"; if (revents _ & POLLNVAL) oss <"NVAL"; return oss. str (). c_str ();}
       
      
     
    
   
  
 

The relationship between the three classes is not hard to understand. In fact, the essence is a Poll/Epoll, but after a higher abstraction, these classes are divided, focus on understanding the class chart at the beginning of the blog.

Refer:

Muduo User Manual

Linux multi-thread Server programming