APR 18 Async + Callback thread Queue Thread Event Association (yield,greenlet,gevent)

APR 18

First, asynchronous + callback mechanism

A, Problem introduction

Problem:

1) The return value of the task cannot be processed in a timely manner, and must wait until all tasks have been completed to be processed uniformly.

2) The process of parsing is executed serially, if parsing takes 2s, parsing 9 times will cost 18s

From concurrent.futures import Processpoolexecutor,threadpoolexecutor

Import OS

Import requests

Import time

Import Random

def get (URL):

Print ('%s GET%s '% (Os.getpid (), URL))

Response=requests.get (URL)

Time.sleep (Random.randint (1,3))

if Response.status_code = = 200:

Return Response.text

def pasrse (res):

Print ('%s ' parsing result:%s '% (Os.getpid (), Len (res)))

if __name__ = = ' __main__ ':

urls=[

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.python.org ',

]

Pool=processpoolexecutor (4)

Objs=[]

For URL in URLs:

Obj=pool.submit (Get,url)

Objs.append (obj)

Pool.shutdown (Wait=true)

For obj in Objs:

Res=obj.result ()

Pasrse (RES)

B, advanced solution: Can solve the above two problems, but make the access information function set and analytic information function Pasrse coupled together

From concurrent.futures import Processpoolexecutor,threadpoolexecutor

Import requests

Import OS

Import time

Import Random

def get (URL):

Print ('%s GET%s '% (Os.getpid (), URL))

Response=requests.get (URL)

Time.sleep (Random.randint (1,3))

if Response.status_code = = 200:

Pasrse (Response.text)

def pasrse (res):

Print ('%s ' parsing result:%s '% (Os.getpid (), Len (res)))

if __name__ = = ' __main__ ':

urls=[

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.python.org ',

]

Pool=processpoolexecutor (4)

For URL in URLs:

Pool.submit (Get,url)

C1, the ultimate solution: can solve the above two problems, while the acquisition of information function set and analytic information function Pasrse decoupling (process version)

The main process acts as the performer of the callback

From concurrent.futures import Processpoolexecutor,threadpoolexecutor

Import requests

Import OS

Import time

Import Random

def get (URL):

Print ('%s GET%s '% (Os.getpid (), URL))

Response=requests.get (URL)

Time.sleep (Random.randint (1,3))

if Response.status_code = = 200:

# Dry parsing of the live

Return Response.text

def pasrse (obj): #后续回调是obj会将自身传给pasrse, so pasrse must have and only one parameter

Res=obj.result ()

Print ('%s ' parsing result:%s '% (Os.getpid (), Len (res)))

if __name__ = = ' __main__ ':

urls=[

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.python.org ',

]

Pool=processpoolexecutor (4)

For URL in URLs:

Obj=pool.submit (Get,url)

Obj.add_done_callback (Pasrse)

Print (' main process ', Os.getpid ())

C2, the ultimate solution: can solve the above two problems, while the acquisition of information function set and the analytic information function Pasrse decoupling (threaded version)

Which child process is idle by that child process as the executor of the callback

From concurrent.futures import Processpoolexecutor,threadpoolexecutor

From threading Import Current_thread

Import requests

Import OS

Import time

Import Random

def get (URL):

Print ('%s GET%s '% (Current_thread (). Name,url))

Response=requests.get (URL)

Time.sleep (Random.randint (1,3))

if Response.status_code = = 200:

# Dry parsing of the live

Return Response.text

def pasrse (obj):

Res=obj.result ()

Print ('%s parsing result:%s '% (Current_thread (). Name,len (res)))

if __name__ = = ' __main__ ':

urls=[

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.baidu.com ',

' Https://www.python.org ',

]

Pool=threadpoolexecutor (4)

For URL in URLs:

Obj=pool.submit (Get,url)

Obj.add_done_callback (Pasrse)

Print (' main thread ', Current_thread (). Name)

Second, thread queue

Import queue

Q=queue. Queue (3) #队列: FIFO

Q.put (1)

Q.put (2)

Q.put (3)

# Q.put (4) #阻塞

Print (Q.get ())

Print (Q.get ())

Print (Q.get ())

Q=queue. Lifoqueue (3) #堆栈: LIFO

Q.put (' a ')

Q.put (' B ')

Q.put (' C ')

Print (Q.get ())

Print (Q.get ())

Print (Q.get ())

Q=queue. Priorityqueue (3) #优先级队列: You can store values in the queue as a small tuple, the first element represents the priority, the smaller the number, the higher the priority

Q.put ((Ten, ' User1 '))

Q.put (( -3, ' user2 '))

Q.put (( -2, ' User3 '))

Print (Q.get ())

Print (Q.get ())

Print (Q.get ())

Third, thread event

A, case one: After waiting for check to reset the value in the event, connect continues to run from event.wait ()

From threading Import Event,current_thread,thread

Import time

Event=event () #event内部维护着一个全局变量

def check ():

Print ('%s ' is detecting service normal .... '%current_thread (). ')

Time.sleep (3)

Event.set () #改变event中的全局变量的值

Def connect ():

Print ('%s waiting to connect ... '%current_thread (). Name)

Event.wait () #等待全局变量的值被重置; If the brackets are 1, that's 1 seconds.

Print ('%s start connection ... '% current_thread (). Name)

if __name__ = = ' __main__ ':

T1=thread (Target=connect)

T2=thread (Target=connect)

T3=thread (Target=connect)

C1=thread (Target=check)

T1.start ()

T2.start ()

T3.start ()

C1.start ()

B, case two: three times after the brush attempt to exit

From threading Import Event,current_thread,thread

Import time

Event=event ()

def check ():

Print ('%s ' is detecting service normal .... '%current_thread (). ')

Time.sleep (5)

Event.set ()

Def connect ():

Count=1

While not Event.is_set ():

if Count = = 4:

Print (' Too many attempts, please retry later ')

Return

Print ('%s attempted connection ... '% (Current_thread (). Name,count))

Event.wait (1)

Count+=1

Print ('%s start connection ... '% current_thread (). Name)

if __name__ = = ' __main__ ':

T1=thread (Target=connect)

T2=thread (Target=connect)

T3=thread (Target=connect)

C1=thread (Target=check)

T1.start ()

T2.start ()

T3.start ()

C1.start ()

Iv.. Co-process

1, single-threaded implementation of concurrency: co-process (in order to improve efficiency, but not that all the process will improve efficiency)

Concurrency refers to multiple tasks that appear to be running concurrently; the nature of the concurrency implementation: Toggle + Save State

The effective co-operation has ' cheated ' the CPU to a certain extent, and through its own internal coordination, it is cut into its own other programs when it encounters Io, which makes the CPU think that the program has been running, which makes it more likely to be in the ready state or run state to occupy more CPU.

2, the implementation of three means of concurrency:

A) concurrency under a single thread, controlled by the program itself, relatively fast

b) concurrency under multithreading, controlled by the operating system, relatively slow

c) Concurrency under multiple processes, controlled by the operating system, relatively slow

3, based on the yield save state, to achieve two tasks to switch directly back and forth, that is, the effect of concurrency (but yield does not encounter blocking automatic cutting program)

PS: If you add a print to each task, it is obvious that the print of two tasks is the one I once, which executes concurrently.

Import time

DEF consumer ():

"Task 1: Receive data, process data"

While True:

X=yield

Def producer ():

"Task 2: Production data"

G=consumer ()

Next (g)

For I in Range (10000000):

G.send (i)

Start=time.time ()

Producer () #1.0202116966247559

Stop=time.time ()

Print (Stop-start)

# purely computed tasks concurrent execution

Import time

Def task1 ():

Res=1

For I in Range (1000000):

Res+=i

Yield

Time.sleep (10000) #yield不会自动跳过阻塞

Print (' Task1 ')

Def task2 ():

G=task1 ()

Res=1

For I in Range (1000000):

Res*=i

Next (g)

Print (' Task2 ')

Start=time.time ()

Task2 ()

Stop=time.time ()

Print (Stop-start)

Five, single-threaded implementation of the IO switch encountered

1, with Greenlet (package yield, encountered IO does not automatically cut)

From Greenlet import Greenlet

Import time

Def eat (name):

Print ('%s eat 1 '%name)

Time.sleep (30)

G2.switch (' Alex ') #只在第一次切换时传值

Print ('%s eat 2 '%name)

G2.switch ()

def play (name):

Print ('%s play 1 '%name)

G1.switch ()

Print ('%s play 2 '%name)

G1=greenlet (EAT)

G2=greenlet (play)

G1.switch (' Egon ')

2, with Gevent module (package Greenlet, do not handle the words, encountered their own IO to actively cut)

Import Gevent

Def eat (name):

Print ('%s eat 1 '%name)

Gevent.sleep (5) #换成time. Sleep (5), does not automatically cut

Print ('%s eat 2 '%name)

def play (name):

Print ('%s play 1 '%name)

Gevent.sleep (3)

Print ('%s play 2 '%name)

G1=gevent.spawn (Eat, ' Egon ')

G2=gevent.spawn (play, ' Alex ')

# gevent.sleep (100)

# G1.join ()

# G2.join ()

Gevent.joinall ([G1,G2])

3, with Gevent module (package Greenlet, processing words, encountered other IO also active cutting)

From gevent import Monkey;monkey.patch_all ()

From threading Import Current_thread

Import Gevent

Import time

Def eat ():

Print ('%s eat 1 '%current_thread (). Name)

Time.sleep (5)

Print ('%s eat 2 '%current_thread (). Name)

Def play ():

Print ('%s play 1 '%current_thread (). Name)

Time.sleep (3)

Print ('%s play 2 '%current_thread (). Name)

G1=gevent.spawn (EAT)

G2=gevent.spawn (play)

# gevent.sleep (100)

# G1.join ()

# G2.join ()

Print (Current_thread (). Name)

Gevent.joinall ([G1,G2])

APR 18 Async + Callback thread Queue Thread Event Association (yield,greenlet,gevent)