Python co-process (gevent)

Co-process, also known as micro-threading, is a lightweight thread of user-state .

The co-process has its own register context and stack. When the schedule is switched, the register context and stack are saved elsewhere, and the previously saved register context and stack are restored when it is cut back. So:

The process can retain the state of the last invocation (that is, a specific combination of all local states), each time the procedure is re-entered, which is equivalent to the state of the last call, in other words: The position of the logical stream at the last departure.

Benefits of the co-process:

No overhead for thread context switching
No need for atomic operation locking and synchronization overhead
Easy switching of control flow and simplified programming model
　　high concurrency + high scalability + low cost : A CPU support for tens of thousands of processes is not a problem. Therefore, it is suitable for high concurrency processing.
Disadvantages:

Unable to take advantage of multicore resources: The nature of the process is a single thread , it can not be a single CPU at the same time multiple cores, the process needs and processes to run on multi-CPU. Of course, most of the applications that we write in the day-out are not necessary, except for CPU-intensive applications.
Blocking (Blocking) operations (such as IO) can block the entire program

Using yield to realize the association process

1 Import Time2 ImportQueue3 4 defConsumer (name):5     Print("--->starting ...")6      whileTrue:7New_baozi =yield8         Print("[%s] is eating Baozi%s"%(name, New_baozi))9         #time.sleep (1)Ten  One defproducer (): A next (Con) - Next (Con2) -n =0 the      whileN < 5: -n + = 1 -         Print("\033[32;1m[producer]\033[0m is making Baozi%s"%N) -  + con.send (n) - con2.send (n) +  A if __name__=='__main__': atcon = consumer ("C1")#Create a Builder object con -Con2 = Consumer ("C2")#Create a Generator object Con2 -p = producer ()#executes the producer function, p is the function return value

Gevent

Gevent is a third-party library that makes it easy to implement concurrent or asynchronous programming through Gevent, and the main pattern used in Gevent is Greenlet, which is a lightweight coprocessor that accesses Python in the form of a C extension module. Greenlet all run inside the main program operating system process, but they are dispatched in a collaborative manner.

Need to install the Gevent library first

Simple test

1  fromGreenletImportGreenlet2 3 deftest1 ():4     Print(12)5Gr2.switch ()#Toggle6     Print(34)7 Gr2.switch ()8 9 deftest2 ():Ten     Print(56) One Gr1.switch () A     Print(78) -  -GR1 =Greenlet (test1) theGR2 =Greenlet (test2) -  - Gr1.switch () -  + ########## Output ########## -12 +56 A34 at78

Another example

Gevent.sleep () Analog IO Blocking

ImportgeventImport Timedeffoo ():Print('Running in Foo', Time.ctime ()) Gevent.sleep (1)      Print('Explicit context Switch to Foo again', Time.ctime ())defBar ():Print('Explicit context to bar', Time.ctime ()) Gevent.sleep (2)    Print('implicit context switch back to bar', Time.ctime ()) Gevent.joinall ([Gevent.spawn (foo), Gevent.spawn (bar),])

Output

 in foo Thu Oct 10:52:5810:52:5810:52:5920 10:53:00 2016

Crawling Web pages

1 Importgevent2 Import Time3  fromGeventImportMonkey4 Monkey.patch_all ()5  fromUrllib.requestImportUrlopen6 7 8 deff (URL):9     Print('GET:%s'%URL)TenRESP =urlopen (URL) Onedata =Resp.read () A     Print('%d bytes received from%s.'%(len (data), URL)) -  -L = ['https://www.python.org/','https://www.yahoo.com/','https://github.com/'] theStart =time.time () - #For URL in L: - #f (URL) -  + #Gevent.joinall ([ - #gevent.spawn (F, ' https://www.pthton.org/'), + #gevent.spawn (F, ' https://www.yahoo.com/'), A #gevent.spawn (F, ' https://github.com/'), at # ]) -  -  - Gevent.joinall ([ -Gevent.spawn (F,'https://www.bilibili.com/'), -Gevent.spawn (F,'http://weibo.com/'), inGevent.spawn (F,'http://www.qq.com/'), - ]) to  + Print(Time.time ()-start)

Gevent under the socket

Server

1 ImportSYS2 ImportSocket3 Import Time4 Importgevent5 6  fromGeventImportsocket, Monkey7 8 Monkey.patch_all ()9 Ten  One defServer (port): As =Socket.socket () -S.bind (('0.0.0.0', port)) -S.listen (500) the  -      whileTrue: -conn, addr =s.accept () - Gevent.spawn (handle_request, conn) +  -  + defHandle_request (conn): A     Try: at          whileTrue: -data = CONN.RECV (1024) -             Print("recv:", data) - conn.send (data) -             if  notData: - Conn.shutdown (socket. SHUT_WR) in                 # Break -  to     exceptException as ex: +         Print(ex) -     finally: the conn.close () *  $ Panax Notoginseng if __name__=='__main__': -Server (8001)

Clinent

1 ImportSocket2 3HOST ='localhost'  #The remote host4PORT = 8001#The same port as used by the server5s =Socket.socket (socket.af_inet, socket. SOCK_STREAM)6 S.connect ((HOST, PORT))7 8 9  whileTrue:Tenmsg = bytes (Input (">>:"), encoding="UTF8") One S.sendall (msg) Adata = S.RECV (1024) -     #print (data) -     Print('Received', str (data,'UTF8')) the  -S.close ()

Python co-process (gevent)