Implementation of loop in gevent source code

The reason why gevent has good performance is mainly due to the encapsulation of libev. Here we will look at the specific implementation of this Part...

Let's take a look at the libev usage. libev defines various events as Watcher, including timing and Io ..

Gevent encapsulates the libev loop and watcher .. This part is written in cython ..

You can see from the gevent code you have previously viewed that all established coroutines have a common parent coroutine, that is, the hub coroutine. It has a loop object, in fact, gevnet manages the running of the entire loop through the hub coroutine...

Loop is written in cython. The code is in core. pyx...

Let's take a look at some of its attribute definitions:

Cdef libev. ev_loop * _ PTR # reference of libev loop cdef public object error_handler cdef libev. ev_prepare _ prepare # This prepare event calls its callback cdef public LIST _ callbacks every time before loop # The linked list cdef libev of all the callbacks suspended on the current loop object. ev_timer _ timer0 # A timer with a timeout value of 0, which is used to make the loop return immediately

For these attributes, the above annotations should be clear... Well, it's really convenient to use cython to write Python extensions...

Now, let's look at the constructor:

# Constructor def _ init _ (self, object flags = none, object default = none, size_t PTR = 0 ): cdef unsigned int c_flags cdef object old_handler = none # This is the callback before registering each event loop. This will be executed before each loop starts. # In fact, the final call of the _ run_callbacks method libev. ev_prepare_init (& self. _ prepare, <void *> gevent_run_callbacks) # ifdef _ Win32 libev. ev_timer_init (& self. _ periodic_signal_checker, <void *> gevent_periodic_signal_check, 0.3, 0.3) # endif # registers the timer callback. In fact, this callback does nothing, the main purpose is to immediately exit the event loop and process the callback libev. ev_timer_init (& self. _ timer0, <void *> gevent_noop, 0.0, 0.0) If PTR: Self. _ PTR = <libev. ev_loop *> PTR else: c_flags = _ flags_to_int (flags) _ check_flags (c_flags) c_flags | = libev. evflag_noenv if default is none: default = true if _ default_loop_destroyed: default = false if default: Self. _ PTR = libev. gevent_ev_default_loop (c_flags) if not self. _ PTR: Raise systemerror ("ev_default_loop (% s) failed" % (c_flags,) # ifdef _ Win32 libev. ev_timer_start (self. _ PTR, & self. _ periodic_signal_checker) libev. ev_unref (self. _ PTR) # endif else: # reference for loop creation. Generally, self is created here. _ PTR = libev. ev_loop_new (c_flags) if not self. _ PTR: Raise systemerror ("ev_loop_new (% s) failed" % (c_flags,) If default or _ syserr_callback is none: set_syserr_cb (self. _ handle_syserr) # Start prepare on the loop. This ensures that the prepare callback is executed before each loop, that is, the callback libev registered on the loop. ev_prepare_start (self. _ PTR, & self. _ prepare) libev. ev_unref (self. _ PTR) self. _ callbacks = [] # initialize the callback linked list

This is quite simple. I will not elaborate on libev. The code first initializes the prepare event and sets its callback to the gevent_run_callbacks method, this method is actually calling the _ run_callbacks method of the current loop object to process all the callback pending on the current loop... That is to say, every time before running the loop's wait, the callback suspended on the current loop will be processed first .. Here, there are a lot of switches in the coroutine...

In addition, we can see that a timer with a timeout value of 0 is initialized. Why is this used? Well, for example, after we run all the callbacks suspended on the current loop, these callbacks may be suspended on the current loop again. To make eventloop understand the returned result and then process these pending callbacks, a timer with a timeout of 0 will be suspended on the current eventloop, the loop can be returned immediately.

The next step is to build the libev loop. Finally, we can see that the ev_prepare_start method is called to start the prepare event...

Well, let's take a look at how the loop object handles the callback hanging on the current loop, that is, the _ run_callbacks method:

# In gevent_run_callbacks, this method is actually called to run all the callbacks. # These Callbacks are executed before each loop operation. cdef _ run_callbacks (Self ): cdef callback CB cdef object callbacks cdef int COUNT = 1000 libev. ev_timer_stop (self. _ PTR, & self. _ timer0) while self. _ callbacks and count> 0: callbacks = self. _ callbacks self. _ callbacks = [] for CB in callbacks: libev. ev_unref (self. _ PTR) gevent_call (self, CB) count-= 1 # a callback may be added when the callback is run, so timing is performed here # because the timed timeout is 0, so let the loop return immediately and you can process the callback if self. _ callbacks: libev. ev_timer_start (self. _ PTR, & self. _ timer0)

In fact, the code is relatively simple. It is nothing more than traversing the callback queue and executing it, and then determining whether a new callback is added. If so, start the timer with a timeout of 0, let the loop return immediately and then process the callback...

Next let's take a look at the run method of loop... This method is called every time in the hub coroutine:

# Run the loop object def run (self, Nowait = false, once = false): check_loop2 (Self) cdef unsigned int flags = 0 if Nowait: Flags | = libev. evrun_nowait if once: Flags | = libev. evrun_once with nogil: # Here, we mainly run the libev loop libev. ev_run (self. _ PTR, flags)

The code here is also simple. It is actually an event loop that runs libev ....

Okay... In fact, basically the loop is about to be mentioned here ....

Next, let's take a look at how the events are encapsulated in the loop... I/O events are used as an example ....

For the creation of I/O watcher, the IO method is generally called on the loop:

# Create I/O watcher # ifdef _ Win32 def io (self, libev. vfd_socket_t FD, int events, ref = true, priority = none): Return io (self, FD, events, ref, priority) # else def io (self, int FD, int events, ref = true, priority = none): Return io (self, FD, events, ref, priority) # endif

There are two types based on different systems, but we only need to care about the following one, so windows will not be considered for the time being... Here we actually constructed an I/O watcher, constructor. The first is the reference of the current loop, and the second is the file descriptor, the third is the event of interest (read or write )...

Let's take a look at the source code definition of Io watcher:

# I/owatcher definition cdef public class io (watcher) [object pygeventioobject, type pygeventio_type]: watcher_base (IO) # some basic attributes are defined through this macro, for example, the watcher reference of libev # This is actually to start watcher def start (self, object callback, * ARGs, pass_events = false): check_loop2 (self. loop) If callback is none: Raise typeerror ('callback must be callable, not none') self. callback = callback if pass_events: Self. ARGs = (gevent_core_events ,) + ARGs else: Self. ARGs = ARGs libev_unref libev. ev_io_start (self. loop. _ PTR, & self. _ watcher) # Start the IO watcher python_incref active pending # ifdef _ Win32 # Io watcher construction def _ init _ (self, loop, libev. vfd_socket_t FD, int events, ref = true, priority = none): If events &~ (Libev. ev__iofdset | libev. ev_read | libev. ev_write): Raise valueerror ('illegal event mask: % R' % events) cdef int VFD = libev. vfd_open (FD) libev. vfd_free (self. _ watcher. FD) # initialize and set the callback to gevent_callback_io # In the callback, the callback of the First Watcher registration will be executed, and the libev registration of the read/write events will be canceled. ev_io_init (& self. _ watcher, <void *> gevent_callback_io, VFD, events) self. loop = loop if Ref: Self. _ flags = 0 else: Self. _ flags = 4 if priority is not none: Libev. ev_set_priority (& self. _ watcher, priority) # else # The structure here is generally considered. The three important parameters are the reference of the loop object, the file descriptor, and the event type, generally, Def _ init _ (self, loop, int FD, int events, ref = true, priority = none): If FD <0: raise valueerror ('fd must be non-negative: % R' % FD) if events &~ (Libev. ev__iofdset | libev. ev_read | libev. ev_write): Raise valueerror ('illegal event mask: % R' % events) # Call the libev method to initialize I/O watcher libev. ev_io_init (& self. _ watcher, <void *> gevent_callback_io, FD, events) self. loop = loop if Ref: Self. _ flags = 0 else: Self. _ flags = 4 if priority is not none: libev. ev_set_priority (& self. _ watcher, priority) # endif property FD: def _ GET _ (Self): Return libev. vfd_get (self. _ watcher. FD) def _ SET _ (self, long FD): If libev. ev_is_active (& self. _ watcher): Raise attributeerror ("'IO 'watcher attribute 'fd' is read-only while watcher is active") cdef int VFD = libev. vfd_open (FD) libev. vfd_free (self. _ watcher. FD) libev. ev_io_init (& self. _ watcher, <void *> gevent_callback_io, VFD, self. _ watcher. events) Property events: def _ GET _ (Self): return self. _ watcher. events def _ SET _ (self, int events): If libev. ev_is_active (& self. _ watcher): Raise attributeerror ("'IO 'watcher attribute 'events' is read-only while watcher is active") libev. ev_io_init (& self. _ watcher, <void *> gevent_callback_io, self. _ watcher. FD, events) Property events_str: def _ GET _ (Self): Return _ events_to_str (self. _ watcher. events) def _ format (Self): Return 'fd = % s events = % s' % (self. FD, self. events_str)

Note that watcher_base (IO) is a macro definition that defines some basic attributes, such as Watcher reference of libev...

From the constructor, we can see that the ev_io_init method of libev is called to initialize the internal I/O watcher reference, and its callback is set to the gevent_callback_io method, in the start method, the ev_io_start method of libev is called to start the watcher and the callback method is recorded... If you have a basic understanding of libev, you can use the above Code to understand the running principle of I/O watcher...

However, it is necessary to talk about the callback function gevent_callback_io, that is, after an I/O event that we are interested in occurs, we will call this method for processing ....

It is defined in Callbacks. h and Callbacks. C... In addition, many similar methods are defined by macro definition, such as gevent_callback_timer... I will not elaborate on this part .. Let's take a look at what code is called:

// First obtain the wather object, and then call the callback # define define_callback (watcher_lc, watcher_type) Static void gevent_callback _ # watcher_lc (struct ev_loop * _ loop, void * c_watcher, int revents) {struct pygevent # watcher_type # object * watcher = get_object (pygevent # watcher_type # object, c_watcher, _ watcher); gevent_callback (watcher-> loop, watcher-> _ callback, Watcher-> ARGs, (pyobject *) watcher, c_watcher, revents );}

First obtain the corresponding watcher object, and then call the gevent_callback method to call the callback registered on watcher...

// For the callback of all events, the static void gevent_callback (struct pygeventloopobject * loop, pyobject * callback, pyobject * ARGs, pyobject * watcher, void * c_watcher, int revents) {gil_declare; pyobject * result, * py_events; long length; py_events = 0; gil_ensure; py_incref (loop); py_incref (callback ); py_incref (ARGs); py_incref (watcher); gevent_check_signals (loop); If (ARGs = py_none) {ARGs = _ pyx_em Pty_tuple;} length = pytuple_size (ARGs); If (length <0) {gevent_handle_error (loop, Watcher); goto end;} If (length> 0 & pytuple_get_item (ARGs, 0) = gevent_core_events) {py_events = pyint_fromlong (revents); If (! Py_events) {gevent_handle_error (loop, Watcher); goto end;} pytuple_set_item (ARGs, 0, py_events);} else {py_events = NULL;} result = pyobject_call (callback, argS, null); // execution callback if (result) {py_decref (result);} else {gevent_handle_error (loop, Watcher); If (revents & (ev_read | ev_write )) {// it can be seen that if a read/write event is registered, it will be canceled, indicating that the registration event/* IO watcher must be rewritten for each read/write event: not stopping it may cause the failing callback To be called repeatedly */gevent_stop (watcher, loop); goto end ;}} if (! Ev_is_active (c_watcher) {/* watcher was stopped, maybe by libev. let's call stop () to clean up * 'callback' and 'args' properties, do py_decref () and ev_ref () if necessary. * BTW, we don't need to check for ev_error, because libev stops the watcher in that case. */gevent_stop (watcher, loop);} end: If (py_events) {py_decref (py_events); pytuple_set_item (ARGs, 0, gevent_core_events);} py_decref (watcher ); py_decref (ARGs); py_decref (callback); py_decref (loop); gil_release ;}

The code of the gevent_callback method is listed here... It is because some of the I/o Event types need to be noted that after the callback is processed, the I/O events suspended on the loop will be canceled...

In this case, you need to register on the loop for each read/write operation .... Well .. But the benefit is .. It provides a paradigm for synchronization...

Now, the implementation of gevent loop is almost the same...

Implementation of loop in gevent source code