Learn more about the co-function in Python

This article to share the content is in-depth understanding of Python in the co-function, has a certain reference value, the need for friends can refer to

Concept:

According to Wikipedia's definition, "a process is a computer program component that produces subroutines for non-preemptive multitasking, allowing different entry points to pause or start executing programs at different locations." From a technical standpoint, "a process is a function that you can pause to execute." If you think of it as "like a generator," then you're right.

The Association, also known as a micro-thread, looks like a subroutine, but it is not the same as the subroutine, it can be executed in the process of breaking the current subroutine after the execution of other subroutines, and then return to execute the previous subroutine, but its related information is the previous.

The process is different from the thread , the thread is preemptive scheduling, and the co-process is a cooperative scheduling, the co-process needs to do its own scheduling.
Subroutine calls are always one entry, one return, and the call order is clear. And the call of the association and subroutine is different. The process appears to be a subroutine, but during execution, it can be interrupted inside the subroutine, then execute another subroutine, and then return to execute at the appropriate time.

Advantages of the co-process:

• The co-process advantage is extremely high execution efficiency. Because the subroutine switch is not a thread switch, but is controlled by the program itself, therefore, there is no thread switching overhead, and multithreading ratio, the more the number of threads, the performance advantage of the association is more obvious. It is appropriate to perform a multi-tasking process.

• There is no thread security issue with the process. A process can have multiple threads at the same time, but only one is active, and the activation and hibernation of the threads is programmed by the programmer to control, not the operating system.

The generator implements the principle of co-process

Example:

def func (n):    index=0    if index<=n:        C=yield 1        print ("task------{}". Format (c))        Index+=1f=func (3) N=next (f) print (n) Try:    n=f.send (5) #程序就直接结束了    print ("N is {}". Format (n)) except Stopiteration as E:    Pass

Output Print: 1task------5

Explanatory notes:

• It is obvious that Func is a generator, and the Send method has a parameter that specifies the return value of the last yield statement that was suspended.

• Send requires exception handling.

• In general, the only difference between the Send method and the next method is that when the Send method is executed, the return value of the last pending yield statement is first set by the parameter, thus interacting with the generator method. However, it is important to note that the Execute send method will cause an error if no yield statement is suspended before a generator object executes the next method.

• When the Send method has a parameter of none, it is completely equivalent to the next method.

Generators implement producer and consumer patterns:

Def Cunsumer (): While    True:        N=yield 3        if not N:            return        print (' cunsumer{} '. Format (n)) def product ( c):    c.send (None)    n=0 while    n<5:        n=n+1        r=c.send (n)        print ("product{}". Format (R))    c.close () C=cunsumer () product (c)

Printed: Cunsumer1product3cunsumer2product3cunsumer3product3cunsumer4product3cunsumer5product3

Explanatory notes:

In the producer first executed the C.send (None), the purpose is to let the consumer hang, and then send value, the first pass 1, the consumer print 1, producer printing R is the value behind consumer yield.

Introduction of Greenlet

Although CPython (standard Python) is able to implement the process through the generator, it is not very convenient to use.

At the same time, a Python-derived version of Stackless Python implements the native co-process, which is more beneficial to use.

So, everyone began to stackless in the code about the process of the separate out to make a CPython expansion package.

This is the origin of Greenlet, so Greenlet is a C extension library that implements the native co-process at the bottom.

CODE schematic:

From Greenlet import greenletimport randomimport timedef Producer (): While    True:        item = random.randint (0,10)        print ("produced {}". Format (item))        C.switch (item) #切换到消费者 and the item is passed to consumer        time.sleep (1) def consumer ():    Print (' I execute first ')    #p. switch () while    True:        item = P.switch () #切换到生产者, and waits for the producer to pass in        the item print (' Consume {} '. format (item)) c = Greenlet (consumer) #将一个普通函数变成一个协程p = Greenlet (Producer) C.switch () #让消费者先进入暂停状态 (only recovered to receive data)

The value of Greenlet:

• High-performance native co-process

• Explicit switching with more explicit semantics

• Wrap functions directly into a process to maintain the original code style

Gevent co-process

Although, we have a epoll-based callback programming pattern, but it is difficult to use.

Even if we can do complex encapsulation with the generator coprocessor, it simplifies the programming difficulty.
But there is still a big problem: The package is difficult, and the existing code is almost completely rewritten

gevent, by encapsulating the Libev (Epoll based) and Greenlet two libraries.
Help us do the encapsulation, allowing us to use the coprocessor in a thread-like way.

So that we can make full use of epoll and coprocessor power without rewriting the original code.

CODE schematic:

From gevent import monkey;monkey.patch_all () #会把python标准库当中一些阻塞操作变成非阻塞import geventdef test1 ():    print ("one")    Gevent.sleep (4) #模拟爬虫请求阻塞    print ("+") def test2 ():    print ("")    Gevent.sleep (4)    print ("44") Gevent.joinall ([Gevent.spawn (Test1), Gevent.spawn (test2)]) #joinall block the current process, execute a given greenlet#spawn start process, and the argument is the name of the function

The value of Gevent:

Switch to another coprocessor to continue execution if you encounter a blocking process!

• Use Epoll-based Libev to avoid blocking.

• Use an efficient gevent-based coprocessor to switch execution.

• Only switches when encountering blocking, no overhead for the wheel, and no thread overhead.

Gevent Implementing concurrent Servers

From gevent import monkey;monkey.patch_all ()  #建议放在首行, some of the blocking operations in the Python standard library become non-blocking import geventimport socketserver= Socket.socket () Server.bind ((", 6666)) Server.listen (5) print (" Start listening ") def readable (con,addr):    print (" Client {} access ". Format (addr)) while    True:        data=con.recv (1024x768)        if data:            print (data)        else:            con.close ()            breakwhile True:    con,addr=server.accept ()    gevent.spawn (readable,con,addr) #将readable函数变为协程, and pass con and addr into them.

Gevent co-process communication

Gevent also has its own queue. Use the same way as the incoming thread.

Producer and consumer patterns based on gevent and queues

From gevent import monkey;monkey.patch_all () Import geventfrom gevent.queue import queueimport randomdef producter (queue ): While    True:        item=random.randint (0,99)        print (' produced {} '. Format (item))        Queue.put (item)        Gevent.sleep (1) def comuser (queue): While    True:        item=queue.get ()        print (' Consuming {} '. Format (item)) queue= Queue () p=gevent.spawn (producter,queue) c=gevent.spawn (comuser,queue) Gevent.joinall ([P,c])

Printing: Produced 33 consumed 33 produced 95 consumed 95 produced 92 consumed 92 ...

Related recommendations:

Use of multi-process + co-routines in Python

Python in co-process

Detailed usage and examples of the Python process