Threads Queue thread pool co-thread

1. Thread queue

From multiprocessing Queue, Joinablequeue #进程IPC队列

From queue import Queue #线程队列 FIFO

From queue import Lifoqueue #后进先出的

The method is the same: put, get, put_nowait, get_nowait, full, empty, qsize

Queues queue: FIFO, self-locking, data security

Stack lifoqueue: LIFO, self-locking, data security

Q = Lifoqueue (5)

Q.put (3)

Q.put_nowait (4)

Print (Q.get ()) #谁最后进的, who will be taken first

Q.get_nowait ()

Print (Q.full ())

Pirnt (Q.empty ())

Print (Q.qsize ())

From queue import Priorityqueue #优先级队列

Q = Priorityqueue ()

Q.put (("AAA"))

Q.put ((4, "BBB"))

Print (Q.get ())

Thread pool

Threading no thread pool.

Multiprocessing Pool

Concurrent.futures help manage thread pool and process pool

Import time

From threading Import Currentthread,get_ident

From concurrent.futures import Threadpoolexecutor #帮助启动线程池

From concurrent.futures import Processpoolexecutor #帮助启动进程池

def func (i):

Time.sleep (1)

Print ("in%s%s"% (I,currentthread ()))

Return i**2

def back (FN):

Print (Fn.result (), CurrentThread ())

#map启动多线程任务

t = Threadpoolexecutor (5)

T.map (Func,range ()) = = = I in range (20):

T.submit (Func,i)

#submit异步提交任务

t = Threadpoolexecutor (5)

For I in range (20):

T.submit (Func,i)

T.shutdown ()

Print ("main:", CurrentThread ()) #起多少个线程池, number of 5*CPU

#获取任务结果

t = Threadpoolexecutor (5)

Li = []

For I in range (20):

ret = T.submit (func,i)

Li.append (ret)

T.shutdown ()

For I in Li:

Print (I.result ())

Print ("main:", CurrentThread ())

#回调函数

t = Threadpoolexecutor (5)

For I in range (20):

T.submit (func,i). Add_done_callback (back)

#回调函数 (Process version)

Import Os,time

From concurrent.futures import Processpoolexecutor

def func (i):

Print ("in%s%s"% (I,os.getpid ()))

Return i**2

def back (FN):

Print (Fn.result (), Os.getpid ())

if __name__ = = "__main__":

p = Processpoolexecutor (5)

For I in range (20):

P.submit (func,i). Add_done_callback (back)

Print ("main:", Os.getpid ())

The multiprocessing module comes with a process pool

The threading module does not have a thread pool.

Concurrent.futures process pool and thread pool: highly encapsulated, unified usage of the process pool/thread pool

Create thread pool/process pool Processpoolexecutor threadpoolexecutor

Ret. Result () Gets the result, and if you want to implement the async effect, you should use the list

Shutdown = = Close + Join Sync control

The Add_done_callback callback function, the parameter received within the callback function is an object that needs to get the return value through result. The callback function for the process pool is still executing in the main process, but the thread pool's callback function is executed in the thread.

Process: The smallest unit of resource allocation, class

Thread: Minimum unit of CPU dispatch, person

CPython threads cannot take advantage of multicore, multithreading cannot take advantage of multicore, and one thread can perform multiple tasks at the same time.

Co-process: can be on the basis of a thread, and then a task to switch between each other. Saves thread-open consumption.

The co-process is dispatched from the level of the Python code, and the normal thread is the smallest unit of CPU scheduling; The scheduling of the association is not done by the operating system.

The previously learned process switches between the two tasks is the generator function: yield

def func ():

Print (1)

x = yield "AAA"

Print (x)

Yield "BBB"

g = func ()

Print (Next (g))

Print (G.send ("* * *"))

The ability to switch between functions---co-processes

DEF consumer ():

While True:

x = Yield

Print (x)

Def producer ():

g = Consumer ()

Next (g) #预激

For I in range (20):

G.send (i)

Producer ()

Yield only switches between programs, no time to re-use any IO operations

Installation of the module:

PIP3 Install the name of the module to be installed

Use the co-process to reduce the time consumed by IO operations

from gevent import Monkey; Monkey.patch_all ()

Import Gevent,time

Def eat ():

Print ("Eat")

Time.sleep (2)

Print ("Finished eating")

Def play ():

Print ("Play")

Time.sleep (1)

Print ("Finished")

G1 = Gevent.spawn (Eat)

G2 = Gevent.spawn (play)

Gevent.joinall ([G1,G2])

Did not execute, why did not execute??? Do you need to open it?

It didn't turn on, but it switched.

Gevent help us do a switch, do switch is conditional, encountered IO to switch

Gevent does not recognize IO operations other than gevent in this module

Use join to block until the process task finishes

Help Gevent to understand blocking in other modules

The From gevent import Monkey;monkey.patch_all () is written before the other modules are imported.

Use the coprocessor to implement the TCP protocol:

Server:

From gevent import Monkey;monkey.patch_all ()

Import Gevent,socket

def func (conn):

While 1:

Conn.send (b "Hello")

Print (CONN.RECV (1024))

SK = Socket.socket ()

Sk.bind (("127.0.0.1", 9090))

Sk.listen ()

While 1:

CONN,ADDR = Sk.accept ()

Gevent.spawn (Func,conn)

Client:

Import socket

From threading Import Thread

DEF client ():

SK = Socket.socket ()

Sk.connect (("127.0.0.1", 9090))

While 1:

Print (SK.RECV (1024))

Sk.send (b "Bye")

For I in range (20):

Thread (target=client). Start ()

4c Concurrent 50000 QPS

5 processes

20 Threads

500 x co-process

The process can significantly improve CPU utilization in a single-core situation

There is no data insecurity in the process, and there is no time overhead for thread switching/creation; At user level when switching, the program does not block the entire thread because one of the tasks in the process goes into a blocking state

Switching of Threads:

Time slices to reduce the efficiency of the CPU

IO can be cut to improve CPU efficiency

Threads Queue thread pool co-thread