Python's co-process

First, what is the co-process

The process is called micro-threading, is a single thread implementation of the concurrency effect of a concept, one sentence explanation is: The process is a user-state lightweight thread, it can be user-defined program to control and dispatch.

Here are two points to note:

1. Python threads are at the kernel level, that is, scheduled by the operating system control (such as single-threaded encounters io or long execution time will be forced to hand over the CPU Execute permissions, switch other threads to run)

2. Once the single-threaded range is enabled, the switch is controlled from the application level (not the operating system) as soon as the IO is encountered, which is independent of efficiency (non-IO switching versus efficiency independent).

Therefore, the use of the process has the following advantages:

1. The transition of the process is program-level switching, with less overhead, the operating system is completely unaware, and therefore more lightweight

2. Concurrency can be achieved within a single thread to maximize CPU utilization

Its disadvantages are:

1. The nature of the process is single-threaded, unable to take advantage of multi-core, can be a program to open multiple processes, each in-process open multiple threads, each line range open the co-

2. The association refers to a single thread, so once the association is blocked, it will block the entire thread

Summary of the characteristics of the process:

　　　1. Concurrency can only be implemented in one single thread

2. No lock required to modify shared data

3. The context stack in the user program that holds multiple control flows

Additional: A co-process encountered IO operation automatically switch to other Io,yield (how to implement detection, Greenlet can not be implemented, the use of the Gevent module (select mechanism))

Second, how to achieve the 1, the use of Greenlet

Need to install before use: PIP3 install Greenlet

 fromGreenletImportGreenletdefEat (name):Print('%s Eat 1'%name) G2.switch ('Egon')    Print('%s Eat 2'%name) G2.switch ()defPlay (name):Print('%s Play 1'%name) G1.switch ()Print('%s Play 2'%name) G1=Greenlet (Eat) G2=Greenlet (play) G1.switch ('Egon')#parameters can be passed in at the first switch, and will not be required at any later time

Simple switching (in the absence of an IO or a duplication of memory space) will slow down the execution of the program

#Sequential ExecutionImport TimedefF1 (): Res=1 forIinchRange (100000000): Res+=IdefF2 (): Res=1 forIinchRange (100000000): Res*=Istart=Time.time () F1 () F2 () Stop=time.time ()Print('run time is%s'% (Stop-start))#10.985628366470337#Toggle fromGreenletImportGreenletImport TimedefF1 (): Res=1 forIinchRange (100000000): Res+=i g2.switch ()defF2 (): Res=1 forIinchRange (100000000): Res*=I G1.switch () Start=time.time () G1=Greenlet (F1) G2=Greenlet (F2) G1.switch () Stop=time.time ()Print('run time is%s'% (Stop-start))#52.763017892837524

Greenlet just provides a convenient way to switch, and when it comes to a task execution, if it encounters Io, it will block in place, but it will not be able to improve efficiency when it encounters automatic IO switching.

The code for these 20 tasks for single-line thread usually has both a computational and a blocking operation, and we can get stuck in the execution of Task 1 o'clock and use the blocking time to perform task 2, which uses the Gevent module.

2, Gevent

First install GEVENT:PIP3 installed Gevent

-Gevent is a third-party library that makes it easy to implement concurrent or asynchronous programming through Gevent, and the main pattern used in Gevent is Greenlet, which is a lightweight coprocessor that accesses Python in the form of a C extension module. Greenlet all run inside the main program operating system process, but they are dispatched in a collaborative manner.

# usage g1=gevent.spawn (func,1,,2,3,x=4,y=5) creates a co-object g1,spawn the first argument in parentheses is the function name, such as Eat, which can be followed by multiple arguments, which can be positional arguments or keyword arguments. Are all passed to the function eat g2=# wait for the end   of the G1 # wait for the G2 to end #  or two-step cooperation step: Gevent.joinall ([g1,g2])g1.value# get func1 return value

-Automatically switch tasks when encountering IO blocking

ImportgeventdefEat (name):Print('%s Eat 1'%name) Gevent.sleep (2)    Print('%s Eat 2'%name)defPlay (name):Print('%s Play 1'%name) Gevent.sleep (1)    Print('%s Play 2'%name) G1=gevent.spawn (Eat,'Egon') G2=gevent.spawn (play,name='Egon') G1.join () G2.join ()#or Gevent.joinall ([g1,g2])Print('Master')

The above example Gevent.sleep (2) simulates an IO block that gevent can identify,

and Time.sleep (2) or other blocking, gevent is not directly recognized by the need to use the following line of code, patching, you can identify the

　　　　From gevent import monkey;monkey.patch_all () must be placed in front of the patched person, such as the Time,socket module

Or we simply remember: To use gevent, you need to put the from Gevent import Monkey;monkey.patch_all () to the beginning of the file

 fromGeventImportMonkey;monkey.patch_all ()ImportgeventImport Timedefeat ():Print('Eat food 1') Time.sleep (2)    Print('Eat Food 2')defPlay ():Print('Play 1') Time.sleep (1)    Print('Play 2') G1=Gevent.spawn (Eat) G2=gevent.spawn (Play_phone) Gevent.joinall ([g1,g2])Print('Master')

3, Gevent synchronous and asynchronous

 fromGeventImportSpawn,joinall,monkey;monkey.patch_all ()Import Timedeftask (PID):"""Some non-deterministic Task"""Time.sleep (0.5)    Print('Task%s done'%pid)defsynchronous (): forIinchRange (10): Task (i)defasynchronous (): G_l=[spawn (Task,i) forIinchRange (10)] Joinall (g_l)if __name__=='__main__':    Print('Synchronous:') synchronous ()Print('Asynchronous:') asynchronous ()#an important part of the above program is to encapsulate the task function into the gevent.spawn of the Greenlet internal thread. The initialized greenlet list is stored in the array threads, which is passed to the Gevent.joinall function, which blocks the current process and executes all the given Greenlet. The execution process will not continue until all greenlet have been executed. 

4, the application of the process: Crawler

 fromGeventImportMonkey;monkey.patch_all ()ImportgeventImportRequestsImport Timedefget_page (URL):Print('GET:%s'%URL) Response=requests.get (URL)ifResponse.status_code = = 200:        Print('%d bytes received from%s'%(Len (response.text), url)) start_time=time.time () Gevent.joinall ([Gevent.spawn (Get_page,'https://www.python.org/'), Gevent.spawn (Get_page,'https://www.yahoo.com/'), Gevent.spawn (Get_page,'https://github.com/'),]) Stop_time=time.time ()Print('run time is%s'% (Stop_time-start_time))

Python's co-process