[Gevent source code analysis] deep analysis of gevent running process

The running process of gevent has been vague. Recently, the source code is slightly obtained and cannot be exclusive.

Gevent is a high-performance network library with the underlying libevent and later libev with the core greenlet. Gevent and eventlet are close. The only difference is that eventlet is implemented by itself, while gevent uses libev. Both of them are widely used. For example, eventlet is used for underlying network communication of openstack, and gevent is used for goagent.

To understand gevent, you must first understand the scheduling process of gevent. gevent has a hub concept, that is, MainThread, which is used to schedule all other greenlet instances (Coroutine ). In fact, hub is also a greenlet, but it is special. We will find that every time we switch from the hub to a greenlet, we will return to the hub, which is the key to gevent. Note: There is no greenlet chain statement in gevent. The greenlet. switch method is registered to the main loop, and the main loop is switched back at the right time.

You may wonder why you need to switch to the hub every time in this mode? I think there are two reasons:

1. The hub is the core of the event-driven system. After each switch to the hub, the events will continue to run cyclically. If it is not available in one greenlet, other greenlets will not be called.

2. Maintaining the relationship between the two is certainly easier than maintaining multiple relationships. Each time we care about the hub and the current greenlet, we do not need to consider the relationship between each greenlet.

Let's see what happened to the simplest gevent. sleep?

Let's first think about how to schedule the simplest sleep (0? According to the above

1. register the current greenlet switch function with the event Loop

2. Switch to the hub and run the main event loop.

def sleep(seconds=0, ref=True):    hub = get_hub()    loop = hub.loop    if seconds <= 0:        waiter = Waiter()        loop.run_callback(waiter.switch)        waiter.get()    else:        hub.wait(loop.timer(seconds, ref=ref))

When seconds is less than or equal to 0, loop. run_callback (waiter. switch) registers the current greenlet switch to the loop and switches it to the hub using waiter. get. Obviously,

After switching to the hub, call the newly registered callback (waiter. switch) to return to the greenlet where just sleep is located.

The children's shoes that are not familiar with Waiter may be a bit vague. Let's take a look at what Waiter is.

>>> result = Waiter()>>> timer = get_hub().loop.timer(0.1)>>> timer.start(result.switch, 'hello from Waiter')>>> result.get() # blocks for 0.1 seconds    'hello from Waiter'

Timer. start (result. switch, 'Hello from Waiter ') we register a s timer to the hub's main cycle, and the callback is result. switch, and then execute result. get (). The process code is as follows:

def get(self):            assert self.greenlet is None, 'This Waiter is already used by %r' % (self.greenlet, )            self.greenlet = getcurrent()            try:                return self.hub.switch()            finally:                self.greenlet = None

Set self. greenlet to the current greenlet, and then switch to the main loop through self. hub. switch (). It is obvious that the result. switch will be called back in the main loop. See the Code:

def switch(self, value=None):            """Switch to the greenlet if one's available. Otherwise store the value."""            greenlet = self.greenlet            assert getcurrent() is self.hub, "Can only use Waiter.switch method from the Hub greenlet"            switch = greenlet.switch            try:                switch(value)            except:                self.hub.handle_error(switch, *sys.exc_info())

Get the saved greenlet, switch to greenlet. switch (), and return to the method we just called reuslt. get. We can also see from the above assert that this is called in the hub.

Through the above analysis, the friends will certainly understand the gevent execution process.

Here is a problem. What if result. switch occurs first? As shown below:

>>> result = Waiter()>>> timer = get_hub().loop.timer(0.1)>>> timer.start(result.switch, 'hi from Waiter')>>> sleep(0.2)>>> result.get() # returns immediatelly without blocking    'hi from Waiter'

I want to be smart. Open hub. py and check the source code. (The above Waiter code is simplified specially ).

Now that we know the gevent running process, let's take a look at what gevent. spawn and join have done?

Gevent. spawn is actually Greenlet. spawn, so gevent. spawn is to create a greenlet and add the greenlet switch () to the hub Main Loop for callback.

class Greenlet(greenlet):    """A light-weight cooperatively-scheduled execution unit."""    def __init__(self, run=None, *args, **kwargs):        hub = get_hub()        greenlet.__init__(self, parent=hub)        if run is not None:            self._run = run        self._start_event = None    def start(self):        """Schedule the greenlet to run in this loop iteration"""        if self._start_event is None:            self._start_event = self.parent.loop.run_callback(self.switch)    @classmethod    def spawn(cls, *args, **kwargs):        """Return a new :class:`Greenlet` object, scheduled to start.        The arguments are passed to :meth:`Greenlet.__init__`.        """        g = cls(*args, **kwargs)        g.start()        return g

The following code proves that:

import geventdef talk(msg):    print(msg)g1 = gevent.spawn(talk, 'bar')gevent.sleep(0)

Output: bar, we switch to the hub through sleep, and then the hub will run the callback talk we added. Everything works normally.

At this time, do not be complacent. If the following code also feels that everything is normal, it is not too late to be happy.

import geventdef talk(msg):    print(msg)    gevent.sleep(0)    print msgg1 = gevent.spawn(talk, 'bar')gevent.sleep(0)

This time it is still output: bar. Something is wrong. We should output two bars. Why is this caused?

Let's take a good look at the process:

1. gevent. spawn registration callback talk

2. Then, the last line gevent. sleep (0) registers the current greenlet. switch (outermost) to the hub, and then switches to the hub

3. The hub executes the callback talk and prints "bar". In this case, gevent. sleep registers g1.switch to the hub again and switches to the hub

4. As the outermost greenlet of step 2 is now registered, the outermost greenlet will be called. Obviously, the program will end. Because the outermost greenlet is not the child greenlet of the hub,

Therefore, after died, it will not return to the parent greenlet, that is, the hub

You may say that I can switch to the hub manually? This will cause the main loop to end.

import geventdef talk(msg):    print(msg)    gevent.sleep(0)    print msgg1 = gevent.spawn(talk, 'bar')gevent.get_hub().switch()

Program output:

barbarTraceback (most recent call last):  File "F:\py_cgi\geve.py", line 9, in <module>    gevent.get_hub().switch()  File "C:\Python26\lib\site-packages\gevent\hub.py", line 331, in switch    return greenlet.switch(self)gevent.hub.LoopExit: This operation would block forever

Although the bar is successfully output twice, it also causes more serious problems.

This is the value of the existence of join. Let's see how join works?

    def join(self, timeout=None):        """Wait until the greenlet finishes or *timeout* expires.        Return ``None`` regardless.        """        if self.ready():            return        else:            switch = getcurrent().switch            self.rawlink(switch)            try:                t = Timeout.start_new(timeout)                try:                    result = self.parent.switch()                    assert result is self, 'Invalid switch into Greenlet.join(): %r' % (result, )                finally:                    t.cancel()            except Timeout:                self.unlink(switch)                if sys.exc_info()[1] is not t:                    raise            except:                self.unlink(switch)                raise    def rawlink(self, callback):        """Register a callable to be executed when the greenlet finishes the execution.        WARNING: the callable will be called in the HUB greenlet.        """        if not callable(callback):            raise TypeError('Expected callable: %r' % (callback, ))        self._links.append(callback)        if self.ready() and self._links and not self._notifier:            self._notifier = self.parent.loop.run_callback(self._notify_links)    def _notify_links(self):        while self._links:            link = self._links.popleft()            try:                link(self)            except:                self.parent.handle_error((link, self), *sys.exc_info())

From the code, we can see that join will save the current greenlet. switch to a queue, register the _ policy_links callback, and switch to the hub. In the _ policy_links callback, the callback previously registered in the queue will be called in turn.

When we call g1.join (), the outermost greenlet. switch will be registered to the queue, and the program will be successfully ended when the callback is completed. Perfect !!!