Python gevent co-process

Brief introduction

• There is no switching overhead. Because the subroutine switch is not a thread switch, but is controlled by the program itself, without the overhead of thread switching, the execution efficiency is high,
• No locking mechanism is required. Because there is only one thread, there is no simultaneous write variable conflict, the control of shared resources in the process is not locked, only need to determine the state is good, so the execution efficiency is much higher than multithreading

Python's support for the process is very limited, and yield in generator can be implemented to some extent.

Yield

The traditional producer-consumer model is a thread-write message, a thread that takes messages, controls the queue through a lock mechanism, and waits, but can be accidentally deadlocked.

If the use of the process, the producer after the production of the message, directly through the yield jump to the consumer to start execution, after the consumer execution, switch back to the producer continued production, high efficiency

Code

Import Timedefconsumer (): R="'     whiletrue:n=yieldRif  notN:return        Print('[CONSUMER] consuming%s ....'%N) R='OK'defProduce (c): C.next () n=0 whileN < 5: N= n + 1Print('[PRODUCER] producing%s ...'%N) R=c.send (n)Print('[PRODUCER] Consumer return:%s\n'%r) c.close ()if __name__=='__main__': C=consumer () produce (c)

Results

[PRODUCER] Producing 1... [CONSUMER] Consuming1.... [PRODUCER] Consumerreturn: 200Ok[producer] Producing2... [CONSUMER] Consuming2.... [PRODUCER] Consumerreturn: 200Ok[producer] Producing3... [CONSUMER] Consuming3.... [PRODUCER] Consumerreturn: 200Ok[producer] Producing4... [CONSUMER] Consuming4.... [PRODUCER] Consumerreturn: 200Ok[producer] Producing5... [CONSUMER] Consuming5.... [PRODUCER] Consumerreturn: $ OK

Analysis

• First call C.next () to start the generator
• Then, once the thing is produced, switch to consumer execution by C.send (n)
• Consumer through yield to the message, processing, and through yield to pass the results back
• Produce get the results of consumer processing, continue to produce the next message

The whole process is unlocked, executed by a thread, producer and consumer writing to complete the task, so called the co-path

Gevent

Python yield provides basic support for the process, but it is not complete. Third-party gevent provides a more complete range of support for Python

Gevent is a third-party library, through the Greenlet implementation of the process, the basic idea is:

When an greenlet encounters an IO operation (such as accessing the network), it automatically switches to the other Greenlet, waits until the IO operation is complete, and then switches back to execution at the appropriate time. Because the IO operation is very time-consuming and often puts the program in a waiting state, with gevent automatically switching the co-process for us, it is guaranteed that there will always be greenlet running, rather than waiting for IO.

Import gevent def f (N):      for inch range (N):        Print  = Gevent.spawn (f, 5= Gevent.spawn (f, 5= Gevent.spawn (f, 5) G1.join () G2.join () G3.join ()

Results

<greenlet at 0x7f7216efbe10:f (5) > 0<greenlet @ 0x7f7216efbe10:f (5) > 1<greenlet at 0x7f7216efbe10:f (5) > 2<greenlet at 0x7f7216efbe10:f (5) > 3<greenlet at 0x7f7216efbe10:f (5) > 4< Greenlet at 0x7f720f54e0f0:f (5) > 0<greenlet @ 0x7f720f54e0f0:f (5) > 1<greenlet at 0x7f720f54e0f0: F (5) > 2<greenlet at 0x7f720f54e0f0:f (5) > 3<greenlet @ 0x7f720f54e0f0:f (5) > 4<greenlet at 0x7f720f54e 190:f (5) > 0<greenlet at 0x7f720f54e190:f (5) > 1<greenlet at 0x7f720f54e190:f (5) > 2<greenlet At 0x7f720f54e190:f (5) > 3<greenlet at 0x7f720f54e190:f (5) > 4

You can see that 3 Greenlet run sequentially instead of alternating

For the Greenlet to run alternately, it can be gevent.sleep() controlled by handing over control

Import gevent def f (N):      for inch range (N):        Print gevent.getcurrent (), I       gevent.sleep (1= Gevent.spawn (f, 3= Gevent.spawn (f, 3  = Gevent.spawn (f, 3) G1.join () G2.join () G3.join ()

Results

<greenlet at 0x7f74e2179e10:f (3) > 0<greenlet @ 0x7f74da7cb0f0:f (3) > 0<greenlet at 0x7f74da7cb190:f (3) > 0<greenlet at 0x7f74e2179e10:f (3) > 1<greenlet at 0x7f74da7cb0f0:f (3) > 1& Lt Greenlet at 0x7f74da7cb190:f (3) > 1<greenlet to 0x7f74e2179e10:f (3) > 2<greenlet at 0x7f74da7cb0f0:f (3) > 2 <greenlet at 0x7f74da7cb190:f (3) > 2

It can be seen that 3 Greenlet are alternately executed

If you change the loop to 1000, let the execution take a long time, and look at the process, you can see that there is only one thread.

Of course, in the actual code, it is impossible to use Gevent.sleep () to switch the co-process, but in the IO operation is, Gevent automatically switch, the reference code is as follows

Importgevent fromGeventImportmonkey; Monkey.patch_all ()ImportUrllib2deff (URL):Print 'GET:%s'%URL resp=urllib2.urlopen (URL) data=Resp.read ()Print '[%d] bytes received from%s\n'%(len (data), URL) gevent.joinall ([Gevent.spawn (F,'http://www.cnblogs.com/kaituorensheng/'), Gevent.spawn (F,'https://www.python.org/'), Gevent.spawn (F,'https://www.baidu.com'),])

Execution results

get:http://www.cnblogs.com/kaituorensheng/Get:https://www.python.org/Get:https://  www.baidu.com[ from https://www.baidu.com[ from http://www.cnblogs.com/ kaituorensheng/[ from https://www.python.org/

You can see that the 3 URL end order is not executed sequentially.

Note

With Gevent, you can get very high concurrency performance, but gevent can only run under Unix/linux and is not guaranteed to install and run properly under Windows.

Since Gevent is based on the IO switchover, the most amazing thing is that the Web App code we're writing does not need to introduce gevent packages or any code, and in the deployment, with a WSGI server that supports gevent, it gains several times faster performance.

Python gevent co-process