[Gevent source code analysis] binding core. pyx to libev cython

Gevent core encapsulates libev and uses cython syntax. If you are interested, you can study it carefully. In fact, libev has Python encapsulation.

Pyev (https://pythonhosted.org/pyev/), but pyev uses C to write extensions, and the code is very complicated. Read the core. pyx code first.

The cython knowledge used by core. pyx.

I. Basic cython knowledge 1. differences between cdef, def, and cpdef: cdef is used to define functions and variables in C, such as cdef int I. Def is generally used to provide Python access. The variables defined by cdef cannot be accessed in the python environment. Either a Def packaging method is provided or cpdef is used. Cpdef is only used to define functions, which is slower than cdef. It is mainly because the class functions defined by cpdef support heavy load and you need to find the virtual function table when calling, cpdef generates functions for both cython and Python calls.
2. Clear type declaration to improve speed and readability, cython recommends that all variables add types, including types in Python. For example, define the list, cdef list result; if the class is available, cdef object P;
3. How to define the class. For public usage, let's take a look at the definition of the class loop provided by core. pyx.

cdef public class loop [object PyGeventLoopObject, type PyGeventLoop_Type]:    cdef libev.ev_loop* _ptr    cdef public object error_handler    cdef libev.ev_prepare _prepare    cdef public list _callbacks    cdef libev.ev_timer _timer0

You may be curious, why is the brackets behind the loop used? Let's take a look at the INT-type py1_bject in Python,

typedef struct {    PyObject_HEAD    long ob_ival;} PyIntObject;

Similarly, we will generate the pygeventloopobject object.

struct PyGeventLoopObject {  PyObject_HEAD  struct __pyx_vtabstruct_6gevent_4core_loop *__pyx_vtab;  struct ev_loop *_ptr;  PyObject *error_handler;  struct ev_prepare _prepare;  PyObject *_callbacks;  struct ev_timer _timer0;}

Pygeventloop_type is the type object used to define a class.

DL_EXPORT(PyTypeObject) PyGeventLoop_Type = {  PyVarObject_HEAD_INIT(0, 0)  __Pyx_NAMESTR("gevent.core.loop"), /*tp_name*/  sizeof(struct PyGeventLoopObject), /*tp_basicsize*/  0, /*tp_itemsize*/  __pyx_tp_dealloc_6gevent_4core_loop, /*tp_dealloc*/  0, /*tp_print*/  0, /*tp_getattr*/  0, /*tp_setattr*/...}

Let's take a look at the public issue. The public here means that this class will be provided to other. c files, so the corresponding. h file will be generated.
Cdef public LIST _ callbacks, which is similar to the function description cpdef, that is, the method for accessing _ callbacks in python is provided.
Ii. Source Code Overview

def __init__(self, object flags=None, object default=None, size_t ptr=0):        pass

Flags: Determine the asynchronous Io model used by the backend, such as "select, epoll", which can be a direct string or number (refer to libev/ev. h)
Default: whether to use the default loop of libev. Otherwise, a new loop is created.
You can use loop. backend to determine whether it is consistent with your settings. Loop. backend_int returns the internal serial number of libev.
For example:

from gevent import coreflag = "select"loop=core.loop(flag)assert loop.backend == flagassert core._flags_to_int(flag) == loop.backend_int

Watcher supported by libev:
All watcher instances are started through start and callback functions are passed.
1. IO:
Loop. io (int fd, int events, ref = true, priority = none)
FD: file descriptor, which can be obtained through sock. fileno ()
Events: Event 1: Read 2: Write 3. read_write

The following two parameters apply to all watcher
Ref: whether to increase the number of mainloop references. It is increased by default. In libev, watcher. Start increases the number of references, and watcher. Stop reduces the number of references. When libev finds that the number of references is 0 and there is no watcher to be monitored, the loop will exit.
Priority: set priority

2. Timer
Loop. Timer (double after, double repeat = 0.0, ref = true, priority = none)
After: How long will it take to start
Repeat: interval between multiple repetitions
You can look at the following small program:

Def F (): Print time. time () print 'eeeee' from gevent. core import loop L = loop () timer = L. timer (2, 3) # Start after 2 seconds, and start print time again after 3 seconds. time () timer. start (f) L. run ()

3. Processing Method of signal received by signer
Loop. Signal (int signum, ref = true, priority = none)
The Hub contains the encapsulated signal, which is used as follows:

def f():    raise ValueError(‘signal‘)sig = gevent.signal(signal.SIGALRM, f)assert sig.ref is Falsesignal.alarm(1)try:    gevent.sleep(2)    raise AssertionError(‘must not run here‘)except ValueError:    assert str(sys.exc_info()[1]) == ‘signal‘

Unlike other watcher, the default ref value is false. Because the signal is not mandatory, the loop does not need to wait for the signal to occur.

4. async wake up the thread
Loop. async (ref = true, priority = none)
This is mainly implemented through the pipeline. The async. Send method will send data to the pipeline and cyclically check the read event to wake up the thread.

    hub = gevent.get_hub()    watcher = hub.loop.async()    gevent.spawn_later(0.1, thread.start_new_thread, watcher.send, ())    start = time.time()    with gevent.Timeout(0.3):        hub.wait(watcher)

Async is used in the thread pool in gevent. When the worker thread runs the callback function, it sets the return value and wakes up the hub main thread through async. Send.

5. Fork subprocess events
Loop. Fork (ref = true, priority = none)
When fork is called, the registered sub-process watcher will be called back, But libev. ev_loop_fork must be called for validity,
To use libev in a sub-process, you must also call libev. ev_loop_fork.
Fork monitor is used in threadpool of gevent

    self.fork_watcher = hub.loop.fork(ref=False)    self.fork_watcher.start(self._on_fork)    def _on_fork(self):        # fork() only leaves one thread; also screws up locks;        # let‘s re-create locks and threads        pid = os.getpid()        if pid != self.pid:            self.pid = pid            # Do not mix fork() and threads; since fork() only copies one thread            # all objects referenced by other threads has refcount that will never            # go down to 0.            self._init(self._maxsize)

The purpose of callback _ on_fork is to re-initialize the thread pool, but we just mentioned that to make the sub-process effective, you must call libev. ev_loop_fork,
Where is this called?

If hasattr (OS, 'fork'): _ fork = OS. fork def fork (): Result = _ fork () if not result: # subprocess reinit () # Call libev. ev_loop_fork return result

The key to the problem is that the fork function is redefined in gevent/OS. py. When fork returns 0, that is, the sub-process, it will call reinit.
Finally, the loop. reinit of core. pyx is called.

    def reinit(self):        if self._ptr:            libev.ev_loop_fork(self._ptr)

The thread pool in gevent is widely used in gevent, especially in windows, such as DNS requests and OS. Read Write are all passed through the thread pool,
Take a moment to look at the threadpool. py source code, you will get a lot.

6. ev_prepare events before each event Loop
Loop. Prepare (ref = true, priority = none)
As mentioned in timeout, callback in the loop has a higher priority than the timer. No callback is added in the loop, and gevent uses
Ev_prepare.
Implementation principle of gevent loop. run_callback:

1. loop. run_callback is directed to loop. _ callbacks: Add callback 2. initialize prepare: libev in _ init _ of loop. ev_prepare_init (& self. _ prepare, <void *> gevent_run_callbacks) register the callback as gevent_run_callbacks 3. in gevent_run_callbacks, The _ run_callbacks result = (struct _ pyx_vtabstruct_6gevent_4core_loop *) loop->__ pyx_vtab)-> _ run_callbacks (loop); 4. callback in _ run_callbacks of loop will be called one by one

This is why the callback has a higher priority.

Loop. run_callback returns a callback object with the stop () and pending attributes. That is to say, if the callback has not been run, we can stop it by the stop () method.
The example code is as follows:

Def F (a):. append (1) From gevent. hub import get_hubloop = get_hub (). loopa = [] f = loop. run_callback (F, a) F. stop () gevent. sleep (0) Assert not F. pending # No blocking may be a running or stopped Assert not

If F. Stop () is commented out, A is [1], because gevent. Sleep (0) is also a direct run_callback, and it must be called first by who joins the program,
However, if it is another watcher, there is no chance to call it.

7. ev_check events after each event Loop
Loop. Check (ref = true, priority = none)
This is the opposite of ev_prepare.

8. Stat file attribute changes
Loop. Stat (path, float interval = 0.0, ref = true, priority = none)
Interval indicates how long will libev start to detect File status changes?

Open two windows, one to run the program, and the other to touch the CS. Log File. Is the File status changed?

Open two windows, one to run the program, and the other to touch CS. log File, whether the file is also a state change hub = gevent. get_hub () filename = 'cs. log 'watcher = hub. loop. stat (filename, 2) # The def F (): Print OS. path. exists (filename) watcher. start (f) gevent. sleep (100)

The Watcher encapsulated in core. pyx is almost all described. Let's take a look at the main loop ev_run of libev.

intev_run (EV_P_ int flags){  do    {      ......    }  while (expect_true (    activecnt    && !loop_done    && !(flags & (EVRUN_ONCE | EVRUN_NOWAIT))  ));  return activecnt;}

Activecnt is the reference count of the loop we mentioned above, so it is best to set ref to true unless in special cases.

Core. pyx details still need to be studied by our friends. The basic usage has been quite clear above. What else does gevent have? Maybe threadpool and DNS will analyze it in detail, so stay tuned.

[Gevent source code analysis] binding core. pyx to libev cython