Python Concurrent Association Process

                                                         "Python Concurrency Association"
One:
  Concurrency is implemented on a single thread, that is, only one main thread, and only one CPU is implemented concurrently. (Nature of Concurrency: Toggle + save State)
  The CPU is running a task, and in both cases it will be cut to perform other tasks (the switchover is controlled by the operating system), in which case the task is blocked and the second case is that the task is taking too long to calculate.

Three states of the main thread:
The second case does not improve efficiency, just to allow the CPU to rain and equitably, to achieve the effect that looks like everyone is executed, if more than one program is purely computational tasks, this switch will reduce efficiency.
For this we are based on yield validation. Yield itself is a way to save the running state of a task in a single thread.
Yiled:
    1: Can save the state, the state of yiled is similar to the state of the operating system's save thread, but the yiled is code-level controlled and more lightweight.
    2:send can pass the result of a function to another function, in order to realize the switch between the single-thread program.

#串行执行import Timedef Consumer (res):    print (' Task 1: Receive data, process data ') def producer ():    print (' Task 2: Generate Data ')    res = [] For    i in range (10000000):        res.append (i)    return resstart = Time.time () #串行执行res = producer () consumer (RES) Stop = Time.time () print (Stop-start) #基于yield并发执行 # import time# def consumer (): #     # print (' Task 1: Receive data, process data ') #     While true:#         x = yield# def producer (): #     # print (' Task 2: Production data ') #     g = consumer () #     Next (g) # for     i in RA Nge (10000000): #2.691828966140747#         G.send (i) # start = Time.time () #基于yiled保存状态, enabling two tasks to switch directly back and forth, that is, concurrent effects # PS: If each task is added to print, then obviously see two tasks are you once I once, that is, concurrent execution # producer () # stop = Time.time () # Print (Stop-start)

In the case of single-threaded, we inevitably appear in the program IO operation, but if we can in our own program (that is, the user program level, not the operating system level) control multiple tasks can encounter IO switch, so that the thread can be maximized in the ready state, that the CPU can be executed at any time, It is equivalent to maximizing the IO operation at the user program level, which seems to have been in the process of computation, with less IO.

The essence of co-integration is to improve efficiency by switching to another task to execute in a single-threaded process, with the user controlling a task when it encounters io blocking.

1: Can control the switch between multiple tasks, save the state of the task before switching (re-run, you can continue based on the paused position)

2: As a 1 supplement: The IO operation can be detected, and a switchover occurs when an IO operation is encountered.

Second: Co-name Introduction (coroutine)

Co-process: is a single-threaded concurrency, also known as micro-threading, fiber.

The process is a user-state lightweight thread (which is controlled by the user program itself)

Emphasize:

1:python threads are at the kernel level, which is controlled by the operating system (single-threaded IO or long execution time will be forced to surrender CPU execution permissions and switch other threads to run)

2: Single-wire range open process, once the IO is encountered, the application level (not the operating system) to control the switchover, in order to improve efficiency (non-IO operation switching mode and efficiency independent)

Compared to the operating system control thread switching, the user in a single-threaded control of the switch process

Advantages:

1: The transition cost of the association is smaller, it belongs to the program-level switch, the operating system is not fully aware, and therefore more lightweight.

2: Concurrency can be achieved within a single thread, maximizing CPU utilization

Disadvantages:

1: The nature of the process is single-threaded, unable to take advantage of multicore, can be a program to open more than one process, each process to open multiple threads, each line range open the association.

2: The association refers to a single thread, so once the association is blocked, it will block the entire process.

Features of the co-process:

1: Concurrency must be implemented in only one single thread.

2: No lock required to modify shared data

3: The user program itself holds multiple control flow context stacks.

4: A co-process encountered IO operation automatically switch to other co-process (how to implement detection io,yiled,greenlet can not be implemented, the use of the Geven module (select mechanism))

Three: Greenlet

If we have 20 tasks within a single thread, the code for each task is divided into two parts: the first half is pure computing and the second half is pure IO.

In order to switch between multiple tasks, the yield generator needs to be initialized one time before the generator, then call send ... Very cumbersome, the use of the Greenlet module can be very simple to achieve these 20 tasks direct switching

Rom greenlet import greenletimport timedef Eat (name):    print ('%s eat 1 '%name)    time.sleep (3)    g2.switch (' Goku ') )    print ('%s eat 2 '%name)    G2.switch () def play (name):    print ('%s play 1 '%name)    g1.switch ()    print (' %s Play 2 '%name) G1 = Greenlet (eat) g2 = Greenlet (play) G1.switch (' Goku ') #可以在第一次swith时传入参数, no need for later
--------------------------Results----------------------------------------

Goku eat 1
Goku Play 1
Goku eat 2
Goku play 2

Simple switching (in the absence of IO or without a memory space) will slow down the execution of the program.

Import Timedef F1 ():    res = 1 for    i in range (100000000):        res+=1def F2 ():    res = 1 for    i in range (1000000 XX):        Res*=istart = Time.time () F1 () f2 () stop = Time.time () print (' Run time is%s '% (Stop-start)) #run time is 23.827917337417603# from Greenlet import greenlet# import time# def F1 (): #     res = # for     i in range (100000000): #
   res+=i#         G2.switch () # def F2 (): #     res = # for     i in range (100000000): #         res*=i#         g1.switch () # Start = Time.time () # G1 = Greenlet (f1) # g2 = Greenlet (F2) # G1.switch () # stop = Time.time () # print (' Run time is%s '% (stop-s Tart)) #158.59044671058655

Greenlet just provides a more convenient way to switch than the generator, when the cut to a task execution encountered blocking IO, then blocking in situ, it is still not resolved to experience the automatic switching of IO to improve efficiency.

Four: Gevent

Gevent is a third-party library that makes it easy to implement concurrent or asynchronous programming through Gevent, and the main pattern used in Gevent is Greenlet, which is a lightweight coprocessor that accesses Python in the form of a C extension module. (Greenlet all run inside the main program OS process, but they are called collaborative scheduling)

Usage:

       G1=gevent.spawn (func,1,,2,3,x=4,y=5) creates a co-object g1,spawn the first argument in parentheses is the function name, such as Eat, which can be followed by multiple arguments, either positional arguments or keyword arguments, which are passed to the       function eat G2=gevent.spawn (FUNC2)       g1.join () #等待g1结束       g2.join () #等待g2结束       #或者上述两步合作一步: Gevent.joinall ([g1,g2])       g1.value# get the return value of func1.
Tasks are automatically switched when an IO block is encountered:

Import Geventdef Eat (name):    print ('%s eat 1 '%name)    gevent.sleep (2)    print ('%s eat 2 '%name) def play (name) :    print ('%s play 1 '%name)    gevent.sleep (3)    print ('%s play 2 '%name) G1 = Gevent.spawn (Eat, ' Goku ') g2= Gevent.spawn (play,name= ' goku ') G1.join () G2.join () #gevent. joinal ([g1,g2]) print (' master ')
--------------------------------------Results-----------------------------------

Goku eat 1
Goku Play 1
Goku eat 2
Goku play 2
Main

The above example Gevent.sleep (2) simulates an IO block that gevent can identify,

and Time.sleep (2) or other blocking, gevent is not directly recognized by the need to use the following line of code, patching, you can identify the

From gevent import Monkey;monkey.patch_all () must be placed in front of the patched person, such as the Time,socket module

Or we simply remember: To use gevent, you need to put the from Gevent import Monkey;monkey.patch_all () to the beginning of the file

From gevent import monkey;monkey.patch_all () Import Geventimport timedef eat ():    print (' eat food 1 ')    time.sleep ( 2)    print (' Eat Food 2 ') def play ():    print (' Play 1 ')    time.sleep (1)    print (' Play 2 ') G1 = Gevent.spawn (Eat) G2 = Gevent.spawn (play) Gevent.joinall ([g1,g2]) print (' master ')
------------------------------------Results----------------------------

At Food 1
Play 1
Play 2
Eat food 2
Main

V: gevent Synchronous asynchronous

From gevent import spawn,joinall,monkey;monkey.patch_all () Import timedef Task (PID):    #一些不确定的任务    Time.sleep ( 0.5)    print (' Task%s done '%pid) def synchronuns (): For    i in range:        Task (i) def asynchronuns ():    g_l = [ Spawn (Task,i) for I in range (Ten)    Joinall (g_l) if __name__ = = ' __main__ ':    print (' synchronous ')    Synchronuns ()    print (' A ')    asynchronuns ()

#上面程序的重要部分是将task函数封装到Greenlet内部线程的gevent. Spawn.
# The  initialized greenlet list is stored in the array threads, this array is passed to the Gevent.joinall function,
# The latter blocks the current process and executes all given Greenlet. The execution process will only be done after all greenlet have been executed.
# will continue to go down.

------------------------------Results-----------------------------------------------

Synchronous
Task 0 Done
Task 1 Done
Task 2 Done
Task 3 Done
Task 4 Done
Task 5 Done
Task 6 Done
Task 7 Done
Task 8 Done
Task 9 Done
A
Task 0 Done
Task 9 Done
Task 8 Done
Task 7 Done
Task 6 Done
Task 5 Done
Task 4 Done
Task 3 Done
Task 2 Done
Task 1 Done

Six: Gevent application Example One
Socket concurrency under single thread via Gevent (from gevent import Monkey;monkey.patch_all () must be placed before the socket module is imported, otherwise gevent will not recognize the blocking of the socket)

Service side

From gevent import Monkey;monkey.patch_all () from socket import *import gevent# If you do not want to patch with Money.patch_all (), You can use Gevent socket# from gevent import socket# s=socket.socket () def server (server_ip,port):    s=socket (Af_inet, SOCK_STREAM)    s.setsockopt (sol_socket,so_reuseaddr,1)    S.bind ((server_ip,port))    S.listen (5)    While true:        conn,addr=s.accept ()        Gevent.spawn (TALK,CONN,ADDR) def talk (conn,addr):    try: While        true :            res=conn.recv (+)            print (' Client%s:%s msg:%s '% (addr[0],addr[1],res))            Conn.send (Res.upper ())    except Exception as E:        print (E)    finally:        conn.close () if __name__ = = ' __main__ ':    server (' 127.0.0.1 ', 8080)

Client

From socket import *client=socket (Af_inet,sock_stream) client.connect ((' 127.0.0.1 ', 8080)) while True:    msg=input ( ' >>: '). Strip ()    if not msg:continue    client.send (Msg.encode (' Utf-8 '))    msg=client.recv (1024x768)    Print (Msg.decode (' Utf-8 '))

Multithreading Concurrent multiple clients

From threading import threadfrom socket import *import threadingdef client (server_ip,port):    c=socket (Af_inet,sock_ STREAM) #套接字对象一定要加到函数内, that is, in the local namespace, outside the function is shared by all threads, then a common socket object, then the client port is always the same    C.connect ((server_ip,port))    Count=0    while True:        c.send (('%s say hello%s '% (Threading.current_thread (). GetName (), count)). Encode (' Utf-8 ')        msg=c.recv (1024x768)        print (Msg.decode (' Utf-8 '))        count+=1if __name__ = = ' __main__ ': for    I In range:        t=thread (target=client,args= (' 127.0.0.1 ', 8080))        T.start ()

Python Concurrent Association Process