Co-processes in the process and Python

Read Catalogue

• 1 co-process
• 2 How to implement a co-process in Python

Back to top 1 the concept of the Coprocessor 1.1 co-process

Co-process, also known as micro-threading, fiber. English name Coroutine. One sentence explains what a thread is: The process is a lightweight thread that is user-state . (In fact, it does not indicate white ~)

I think that the association process, more abstract, if the thread has a certain understanding, it should be better understood.

　　So it's easier to understand the process:

Threads are system-level, they are scheduled by the operating system, and the process is program-level, which is dispatched by the programmer on demand. We call one of the functions in a thread called subroutines, then the subroutine can be interrupted in the execution of the other subroutine, the other subroutine can also be interrupted back to continue to execute the previous subroutine, this is the process. That is, the same thread of code <1> execution can be interrupted, and then jump to execute another piece of code, when again back to execute the code block <1>, and then from the place where the previous break began.

　　The comparative professional understanding is:

The co-process has its own register context and stack. When the schedule is switched, the register context and stack are saved elsewhere, and the previously saved register context and stack are restored when it is cut back. Thus: The process can retain the state of the last invocation (that is, a specific combination of all local states), each time the procedure is re-entered, which is equivalent to the state of the last call, in other words: The position of the logical flow at the last departure.

1.2 Advantages and disadvantages of the co-process

Advantages of the co-process:

(1) without the overhead of thread context switching, the process avoids meaningless scheduling, which can improve performance (but therefore, programmers have to assume responsibility for scheduling, while the process also loses the ability of standard threads to use multiple CPUs)

(2) Cost of locking and synchronizing without atomic operation

(3) Easy to switch the control flow, simplify the programming model

(4) High concurrency + high scalability + Low cost: One CPU support for tens of thousands of processes is not a problem. Therefore, it is suitable for high concurrency processing.

Disadvantages of the co-process:

(1) Unable to take advantage of multi-core resources: The nature of the process is a single-threaded, it can not be a single CPU at the same time multiple cores, the process needs and processes to run on multi-CPU. Of course, most of the applications we write on a daily basis are not necessary, except for CPU-intensive applications.

(2) blocking (Blocking) operation (e.g. IO) blocks the entire program

Back to top 2 how to implement the co-2.1 yield implementation process in Python

As described earlier, "subroutines (functions) can be interrupted in the execution of other subroutines, and other subroutines can be interrupted to continue to execute the previous subroutine," then it is easy to think of Python yield, obviously yield can achieve this switch.

Example of using yield to implement a co-process:

1 #! /usr/bin/env Python 2 #-*-coding:utf-8-*-3 # Author: "Zing-p" 4 # DATE:2017/5/12 5  6  7 def Consumer (name): 8< C2/>print ("To start chew bones ...") 9 while     true:10         print ("\033[31;1m[consumer]%s\033[0m"% name) one         bone = yield12         print ("[%s] is chew bones%s"% (name, bone)) Producer Def (Obj1, obj2):     obj1.send (None)    # launch Obj1 this generator, first time must Must use None  <==> obj1.__next__ ()     obj2.send (none)    # start Obj2 This generator, the first time must be used with none  <==> obj2._ _next__ ()     n = 019 while     n < 5:20         n + = 121         print ("\033[32;1m[producer]\033[0m is producing bone%s"% n) 22
   obj1.send (n)         obj2.send (n) If __name__ = = ' __main__ ': con1     = consumer ("Consumer a")     Con2 = Consumer ("Consumer B")     producer (Con1, Con2)

Results of the operation:

2.2 Greenlet implementation co-process

Python's greenlet is equivalent to manually switch, to execute other sub-programs, in the "other subroutines" and actively switch back ...

Examples of Greenlet process:

1 #! /usr/bin/env Python 2 #-*-coding:utf-8-*-3  4 from Greenlet import Greenlet 5 # Greenlet is actually a manual switch; Gevent is the Greenlet Package, can be implemented automatically switch 6  7 def test1 (): 8     print ("123") 9     Gr2.switch ()   # Switch to execute test210     print ("456")     one by one Gr2.switch ()   # Switch back to the position that you performed before Test2, then execute the test2 ():     print ("789")     Gr1.switch ()   # Switch back to the position that you performed before Test1, and then perform     a print ("666") of Gr1 = Greenlet (test1)   # Start a co-process note test1 do not add () GR2 = Greenlet ( test2)   #21 Gr1.switch ()

2.3 Gevent Implementation co-process

Gevent is a third-party library that can easily implement regulation regulation through gevent, the main pattern used in Gevent is Greenlet, which is a lightweight coprocessor that accesses Python in the form of a C extension module. Greenlet all run inside the main program operating system process, but they are dispatched in a collaborative manner.

Gevent will actively identify the IO operation inside the program, and when the subroutine encounters io, switch to another subroutine. If all of the subroutines go into IO, they are blocked.

Gevent examples of co-threads:

1 #! /usr/bin/env Python3 2 #-*-coding:utf-8-*-3  4 Import gevent 5  6 def func1 (): 7     print ("func1 running") 8
   gevent.sleep (2)             # internal function Implementation IO Operation 9     print ("Switch Func1") def func2 ():     print ("Func2 running")     Gevent.sleep (1)     print ("Switch Func2") def func3 ():     print ("func3  running")     Gevent.sleep (0)     print ("func3 done..") Gevent.joinall ([Gevent.spawn (FUNC1),                 gevent.spawn (FUNC2), Gevent.spawn                 (func3)                 ])

Synchronization and asynchronous performance differences:

1 Import gevent 2   3 def task (PID): 4     "" "5     Some non-deterministic task 6     " "7     Gevent.sleep (0.5) 8
   
    print (' Task%s done '% pid) 9  def synchronous (): one for     I in range (1,10):         Task (i)  def Asynchrono US ():     threads = [Gevent.spawn (task, I) for I in range (Ten)]16     gevent.joinall (threads) +  print (' Synchronous: ') synchronous ()  print (' Asynchronous: ') + asynchronous ()
   

An important part of the above program is to encapsulate the task function into the Greenlet internal thread gevent.spawn . The initialized greenlet list is stored in the array threads , which is passed to the gevent.joinall function, which blocks the current process and executes all the given Greenlet. The execution process will not continue until all greenlet have been executed.

"Crawler" Switching tasks when encountering IO blocking

0 B!  /usr/bin/env Python3 2 #-*-coding:utf-8-*-3  4 from urllib import request 5 import Gevent,time 6 from gevent Import Monkey 7  8 Monkey.patch_all ()    # mark All IO operations in the current program 9 def spider (URL):     print ("get:%s"% url)-     resp = Request.urlopen (URL)     data = Resp.read ()     print ("%s bytes received from%s:"% (len (data), URL)) URLs = ["     https://www.python.org/", "     https://www.yahoo.com/", "https://github.com/"     ]21 Start_  Time = Time.time () for the URL in urls:24     spider (URL) print ("Sync Duration:", time.time ()-start_time) Async_time_start = Time.time () Gevent.joinall ([     Gevent.spawn (Spider, "https://www.python.org/"),     Gevent.spawn ( Spider, "https://www.yahoo.com/"),     Gevent.spawn (spider, "https://github.com/"), + ("Asynchronous time-consuming:", Time.time ()-Async_time_start) 34 35 # It's best to crawl a foreign site.

Multi-socket concurrency under " single thread " with gevent implementation

Server side:

1 Import SYS 2 import socket 3 Import time 4 import gevent 5   6 from gevent import Socket,monkey 7 monkey.patch_all () 8  Ten def server (port): one     s = socket.socket ()     s.bind ((' 0.0.0.0 ', port))-     S.listen (     True:15         cli, addr = s.accept ()         gevent.spawn (handle_request, CLI),  def handle_request (conn): 20     try:21         while true:22             data = CONN.RECV (1024x768)             print ("recv:", data)             conn.send (data) 25             if not data:26                 conn.shutdown (socket. SHUT_WR)     except Exception as  ex:29         print (ex),     finally:31         conn.close () + if __ name__ = = ' __main__ ':     server (9999)

Client side:

1 Import Socket 2   3 host = ' localhost '    # The remote host 4 port = 9999         # The same port as used by the server 5 s = socket.socket (socket.af_inet, socket. SOCK_STREAM) 6 S.connect ((HOST, PORT)) 7 while true:8     msg = bytes (input (">>:"), encoding= "UTF8") 9     S.sendall (msg)     data = S.RECV (1024x768) One-     #print (data)     print (' Received ', repr (data)) S.close ()

Co-processes in the process and Python