Python co-process learning

Co-process

Co-process, also known as micro-threading, fiber. English name Coroutine. One sentence explains what a thread is: The process is a lightweight thread that is user-state.

The co-process has its own register context and stack. When the schedule is switched, the register context and stack are saved elsewhere, and the previously saved register context and stack are restored when it is cut back. So:

The process can retain the state of the last invocation (that is, a specific combination of all local states), each time the procedure is re-entered, which is equivalent to the state of the last call, in other words: The position of the logical stream at the last departure.

Benefits of the co-process:

1. The overhead of no thread context switching
2. No need for atomic operation locking and synchronization overhead
3. Easy to switch the control flow, simplify the programming model
4. High concurrency + high scalability + Low cost: A CPU support for tens of thousands of processes is not a problem. Therefore, it is suitable for high concurrency processing.

Disadvantages:

Unable to take advantage of multicore resources: The nature of the process is a single thread, it can not be a single CPU at the same time multiple cores, the process needs and processes to run on multi-CPU. Of course, most of the applications that we write in the day-out are not necessary, except for CPU-intensive applications.
Blocking (Blocking) operations (such as IO) can block the entire program

Greenlet

From Greenlet import Greenlet

def test1 ():
Print (1)
Gr2.switch ()
Print (4)
Gr2.switch ()
Def test2 ():
Print (2)
Gr3.switch ()
Print (5)
Gr3.switch ()
def test3 ():
Print (3)
Gr1.switch ()
Print (6)
Gr1 = Greenlet (test1)
GR2 = Greenlet (test2)
GR3 = Greenlet (test3)
Gr1.switch ()


Gevent

Gevent is a third-party library that makes it easy to implement concurrent or asynchronous programming through Gevent, and the main pattern used in Gevent is Greenlet, which is a lightweight coprocessor that accesses Python in the form of a C extension module. Greenlet all run inside the main program operating system process, but they are dispatched in a collaborative manner.

Import Gevent

def test1 ():
Print (1)
Gevent.sleep (2)
Print (4)
Def test2 ():
Print (2)
Gevent.sleep (1)
Print (5)
def test3 ():
Print (3)
Gevent.sleep (0)
Print (6)
Gevent.joinall ([
Gevent.spawn (Test1),
Gevent.spawn (Test2),
Gevent.spawn (TEST3),
])

Performance differences between synchronous and asynchronous

Import Gevent

def task (PID):
"""
Some non-deterministic Task
"""
Gevent.sleep (0.5)
Print (' Task%s done '% pid)

def synchronous ():
For I in Range (1,10):
Task (i)

def asynchronous ():
threads = [Gevent.spawn (task, I) for I in range (10)]
Gevent.joinall (Threads)

Print (' Synchronous: ')
Synchronous ()

Print (' Asynchronous: ')
Asynchronous ()

An important part of the above program is to encapsulate the task function into the gevent.spawn of the Greenlet internal thread. The initialized greenlet list is stored in the array threads, which is passed to the Gevent.joinall function, which blocks the current process and executes all the given Greenlet. The execution process will not continue until all greenlet have been executed.

Automatically switch tasks when IO blocking is encountered

from gevent import Monkey; Monkey.patch_all ()
Import Gevent
From urllib.request import Urlopen

def f (URL):
Print (' GET:%s '% URL)
resp = urlopen (URL)
data = Resp.read ()
Print ('%d bytes received from%s. '% (len (data), URL))

Gevent.joinall ([
Gevent.spawn (F, ' https://www.python.org/'),
Gevent.spawn (F, ' https://www.yahoo.com/'),
Gevent.spawn (F, ' https://github.com/'),
])

Multi-socket concurrency with single-threaded implementation via Gevent

Server Side

Import Sys
Import socket
Import time
Import Gevent

From gevent import Socket,monkey
Monkey.patch_all ()

def server (port):
s = Socket.socket ()
S.bind (' 0.0.0.0 ', port)
S.listen (500)
While True:
CLI, addr = s.accept ()
Gevent.spawn (Handle_request, CLI)

DEF handle_request (conn):
Try
While True:
data = CONN.RECV (1024)
Print ("recv:", data)
Conn.send (data)
If not data:
Conn.shutdown (socket. SHUT_WR)

Except Exception as ex:
Print (ex)
Finally
Conn.close ()
if __name__ = = ' __main__ ':
Server (8001)

Client Side

Import socket

host = ' localhost ' # The remote host
Port = 8001 # The same port as used by the server
s = socket.socket (socket.af_inet, socket. SOCK_STREAM)
S.connect ((HOST, PORT))
While True:
msg = bytes (input (">>:"), encoding= "UTF8")
S.sendall (msg)
data = S.RECV (1024)
#print (data)

Print (' Received ', repr (data))
S.close ()

Python co-process learning