Python Concurrent Programming Association Process

Reference: http://www.cnblogs.com/Eva-J/articles/8324673.html

introduction of co-process

Co-process: is a single-threaded concurrency, also known as micro-threading, fiber. English name Coroutine. One sentence describes what a thread is: The process is a lightweight thread of user-state, that is, the process is scheduled by the user program itself. ,

It should be emphasized that:

#1. The python thread is at the kernel level, which is the #2 that is controlled by the operating system (such as a single-threaded encounter with IO or an excessive execution time that is forced to surrender CPU execution permissions and switch other threads to run). The single-wire range opens the process, and once the IO is encountered, the switch is controlled from the application level (not the operating system) to increase efficiency (!!!). Non-IO operation switching is not efficiency-independent)

Compared to the operating system control thread switching, the user in a single-threaded control of the switch process

The advantages are as follows:

#1. The switch overhead of the process is smaller, it is program-level switching, the operating system is completely unaware, and therefore more lightweight. Concurrency can be achieved within a single thread, maximizing CPU utilization

Disadvantages are as follows:

#1. The nature of the process is single-threaded, unable to take advantage of multi-core, can be a program to open more than one process, each process to open multiple threads, each line range open the co-thread. The association refers to a single thread, so once the association is blocked, it will block the entire thread

Summary of the characteristics of the process:

1. Concurrency must be implemented in only one single thread
2. No lock required to modify shared data
3. The context stack in the user program that holds multiple control flows
4. Additional: A co-process encountered IO operation automatically switch to other Io,yield (how to implement detection, Greenlet can not be implemented, the use of the Gevent module (select mechanism))

Greenlet Module

Installation: PIP3 Install Greenlet

G1=gevent.spawn (func,1,,2,3,x=4,y=5) creates a co-object g1,spawn the first argument in parentheses is the function name, such as Eat, which can be followed by multiple arguments, either positional arguments or keyword arguments, which are passed to the eat of the function G2 =# wait for G1 to end  # wait for G2 to end #  or two-step cooperation step: Gevent.joinall ([g1,g2])g1.value# get func1 return value

Usage Introduction

ImportgeventdefEat (name):Print('%s Eat 1'%name) Gevent.sleep (2)    Print('%s Eat 2'%name)defPlay (name):Print('%s Play 1'%name) Gevent.sleep (1)    Print('%s Play 2'%name) G1=gevent.spawn (Eat,'Egon') G2=gevent.spawn (play,name='Egon') G1.join () G2.join ()#or Gevent.joinall ([g1,g2])Print('Master')

Example: Encountering an IO active switch

The above example Gevent.sleep (2) simulates an IO block that gevent can identify, and Time.sleep (2) or other blocking, gevent is not directly recognized by the need to use the following line of code, patching, you can identify the

From gevent import Monkey;monkey.patch_all () must be placed in front of the patched person, such as the Time,socket module

Or we simply remember: To use gevent, you need to put the from Gevent import Monkey;monkey.patch_all () to the beginning of the file

 fromGeventImportMonkey;monkey.patch_all ()ImportgeventImport Timedefeat ():Print('Eat food 1') Time.sleep (2)    Print('Eat Food 2')defPlay ():Print('Play 1') Time.sleep (1)    Print('Play 2') G1=Gevent.spawn (Eat) G2=Gevent.spawn (play) Gevent.joinall ([g1,g2])Print('Master')

View Code

We can use Threading.current_thread (). GetName () to view each G1 and G2, viewing the result as dummythread-n, which is a dummy thread

 fromGeventImportMonkey;monkey.patch_all ()ImportThreadingImportgeventImport Timedefeat ():Print(Threading.current_thread (). GetName ())Print('Eat food 1') Time.sleep (2)    Print('Eat Food 2')defPlay ():Print(Threading.current_thread (). GetName ())Print('Play 1') Time.sleep (1)    Print('Play 2') G1=Gevent.spawn (Eat) G2=Gevent.spawn (play) Gevent.joinall ([g1,g2])Print('Master')

View Threading.current_thread (). GetName ()

 fromGeventImportSpawn,joinall,monkey;monkey.patch_all ()Import Timedeftask (PID):"""Some non-deterministic Task"""Time.sleep (0.5)    Print('Task%s done'%pid)defSynchronous ():#Sync     forIinchRange (10): Task (i)defAsynchronous ():#AsynchronousG_l=[spawn (Task,i) forIinchRange (10)] Joinall (g_l)Print(' Done')    if __name__=='__main__':    Print('Synchronous:') synchronous ()Print('Asynchronous:') asynchronous ()#an important part of the above program is to encapsulate the task function into the gevent.spawn of the Greenlet internal thread. #The initialized greenlet list is stored in the array threads, and this array is passed to the Gevent.joinall function.#The latter blocks the current process and performs all the given Greenlet tasks. The execution process will not continue until all greenlet have been executed. 

gevent Synchronous and asynchronousAn example of application of gevent

 fromGeventImportMonkey;monkey.patch_all ()ImportgeventImportRequestsImport Timedefget_page (URL):Print('GET:%s'%URL) Response=requests.get (URL)ifResponse.status_code = = 200:        Print('%d bytes received from%s'%(Len (response.text), url)) start_time=time.time () Gevent.joinall ([Gevent.spawn (Get_page,'https://www.python.org/'), Gevent.spawn (Get_page,'https://www.yahoo.com/'), Gevent.spawn (Get_page,'https://github.com/'),]) Stop_time=time.time ()Print('run time is%s'% (Stop_time-start_time))

application of the process: CrawlerGevent Application Example Two

Socket concurrency via gevent for single-threaded threads

Note:the From gevent import Monkey;monkey.patch_all () must be placed before the socket module is imported, otherwise gevent does not recognize the blocking of the socket

 fromGeventImportMonkey;monkey.patch_all () fromSocketImport*Importgevent#If you do not want to use Money.patch_all () to patch, you can use the gevent to bring your own socket#From gevent Import socket#S=socket.socket ()defServer (server_ip,port): s=socket (af_inet,sock_stream) s.setsockopt (sol_socket,so_reuseaddr,1) S.bind ((Server_ip,port)) S.listen (5)     whiletrue:conn,addr=s.accept () gevent.spawn (TALK,CONN,ADDR)defTalk (conn,addr):Try:         whileTrue:res=CONN.RECV (1024)            Print('Client%s:%s msg:%s'% (addr[0],addr[1],res)) Conn.send (Res.upper () )exceptException as E:Print(e)finally: Conn.close ()if __name__=='__main__': Server ('127.0.0.1', 8080)

Server

 fromSocketImport*Client=socket (Af_inet,sock_stream) client.connect (('127.0.0.1', 8080)) whiletrue:msg=input ('>>:'). Strip ()if  notMsgContinueclient.send (Msg.encode ('Utf-8')) msg=CLIENT.RECV (1024)    Print(Msg.decode ('Utf-8'))

Client

 fromThreadingImportThread fromSocketImport*ImportThreadingdefClient (Server_ip,port): C=socket (Af_inet,sock_stream)#The socket object must be added to the function, that is, within the local namespace, in the outside of the function is shared by all threads, then everyone common one socket object, then the client port is always the sameC.connect ((server_ip,port)) Count=0 whileTrue:c.send (('%s Say hello%s'% (Threading.current_thread (). GetName (), count)). Encode ('Utf-8')) msg=C.RECV (1024)        Print(Msg.decode ('Utf-8')) Count+=1if __name__=='__main__':     forIinchRange (500): T=thread (target=client,args= ('127.0.0.1', 8080) ) T.start ()

Multithreading concurrent multiple clients

Python Concurrent Programming Association Process