Python Enhanced Generator-coroutine

This article mainly introduces enhanced generator in Python, namely coroutine related content, including basic syntax, usage scenarios, considerations, and similarities and differences with other language implementation.

Enhanced generator

The usage scenarios and ideas of yield and generator are described above, and only the next method of generator is used, in fact generator has more powerful functions. PEP 342 adds a series of methods to generator to make generator more like a co-coroutine. The main change is that early yield can only return a value (as the creator of the data), and the new send method can consume a value when the generator is restored. The caller (generator call) can also be thrown by throwing an exception in the generator pending.

First look at the yield of the enhanced version, the syntax format is as follows:
Back_data = Yield Cur_ret

This code means: When executed to this statement, return Cur_ret to the caller, and when generator through the next () or send (Some_data) method Restore, Some_data assigned to Back_data. For example:

1 def gen (data): 2     print ' before yield ', data 3     back_data = Yield data 4     print ' After Resume ', Back_data 5
   
    6 if __name__ = = ' __main__ ': 7     g = Gen (1) 8     print G.next () 9     try:10         g.send (0) one     except Stopiteration :         Pass
   

Output:
Before yield 1
1
After resume 0

two points to note :

(1) Next () equivalent to send (None)
(2) on the first call, you need to use the next () statement or send (None), you cannot use Send to send a value other than None, otherwise it will be wrong, because there is no Python yield statement to receive this value.

Application Scenarios

Generator has the ability to consume data (push) when generator can accept data (when recovering from a suspended state) rather than just returning data. The following example is from here:

1 word_map = {} 2 def consume_data_from_file (file_name, Consumer): 3 for line in     file (file_name): 4         consumer.send (  Line) 5  6 def consume_words (consumer): 7 and     true:8 line         = yield 9 for         word in (W for W in Line.split () If W.strip ()):             consumer.send (word) All-in-one Def Count_words_consumer (): While     true:14         word  = yield15         if Word not in word_map:16             Word_map[word] = 017         Word_map[word] + = 118     print word_map19 if __name__ = = ' __main__ ':     cons = Count_words_consumer ()     cons.next ()     Cons_inner = Consume_words (cons) 24     Cons_inner.next ()     c = consume_data_from_file (' test.txt ', Cons_inner)-     print Word_map

In the above code, the real data consumer is count_words_consumer, the most primitive data producer is consume_data_from_file, the flow of data is the initiative from the producer to the consumer. However, the 22nd and 24 lines above are called two times next, which can be encapsulated using a decorator.

1 def Consumer (func): 2     def wrapper (*args,**kw): 3         gen = func (*args, **kw) 4         gen.next () 5         return GEN6     wrapper.__name__ = func.__name__7     wrapper.__dict__ = func.__dict__8     wrapper.__doc__  = func.__doc__ 9     return Wrapper

The modified code :

Example_with_deco

Generator throw

In addition to the next and send methods, generator also provides two practical methods, throw and close, which reinforce caller's control of generator. The Send method can pass a value to the Generator,throw method to throw an exception where generator hangs, and the Close method allows the generator to end normally (after which you can no longer invoke next send). The throw method is described in detail below.

throw (type[, value[, Traceback])
Throws a type of exception at generator yield and returns the next yield value. If the type of an exception is not captured, it is passed to caller. Also, if generator cannot yield a new value, the Stopiteration exception is thrown to caller

1 @consumer 2 def gen_throw (): 3     value = Yield  4     try:5         yield value 6     except Exception, E:7         yield STR (e) # If commented out this line, then will throw Stopiteration 8  9 If __name__ = = ' __main__ ':     51 g = Gen_throw () One     assert G.send (5 2     assert G.throw (Exception, ' throw Exception ') = = ' Throw Exception '

The first call to send, the code returns value (5) after the 5th line hangs, and then generator throw will be caught by the 6th row. If line 7th is not re-yield, the stopiteration exception is re-thrown.

　

Precautions

If a generator has already been executed through send, it cannot be dispatched from another generator to the generator until it is again yield

1 @consumer 2 def funca (): 3     while true:4         data = yield 5         print ' Funca recevie ', data 6         fb.send (data * 2) 7  8 @consumer 9 def FUNCB (): Ten while     true:11         data = yield12         print ' FUNCB recevie ', Data13         fa. Send (Data * 2), fa = Funca () + fb = FUNCB () if __name__ = = ' __main__ ':     fa.send (10)

Output:

Funca Recevie 10
FUNCB Recevie 20
Valueerror:generator already executing

Generator and Coroutine

Back to Coroutine, see the Wikipedia explanation (Https://en.wikipedia.org/wiki/Coroutine#Implementations_for_Python), and my own understanding is relatively simple (or one-sided): programmers can control the concurrency process , whether it is a process or a thread, its switch is the operating system in the scheduling, and for the co-process, the programmer can control when to switch out, when to switch back. The progression is much lighter than the process thread, with less overhead for context switching. In addition, because it is the programmer to control the scheduling, to a certain extent can also avoid a task interrupted by the middle. Which scenarios can be used in the process, I think can be summed up as non-blocking waiting for the scene, such as game programming, asynchronous Io, event-driven.

In Python, generator's send and throw methods make generator very much like a co-process (coroutine), but generator is just a semi-association (semicoroutines), as Python doc describes:

"All of the makes generator functions quite similar to coroutines; They yield multiple times, they has more than one entry point and their execution can be suspended. The only difference are a generator function cannot control where should the execution continue after it yields; t He control is always transferred to the generator ' s caller."

However, even more powerful functions can be achieved with enhanced generator. For example, the Yield_dec mentioned above can only passively wait until the time arrives and then continue execution. In some cases, such as triggering an event, we want to resume the execution process immediately, and we are concerned about what the event is, and we need to send it at generator. In another case, we need to terminate this execution process, then deliberately call close, while in the code to do some processing, pseudo-code as follows:

1 @yield_dec2 def do (a): 3     print ' do ', A4     try:5         event = yield,         print ' Post_do ', A, event7     finally:8< C5/>print ' Do sth '

As for another example mentioned earlier, asynchronous invocations between services (processes) are also examples that are well suited to practical processes. Callback's way of splitting code, dividing a piece of logic into multiple functions, is a much better way to do it, at least for code reading. Other languages, such as C #, go language, are standard implementations, especially for the Go language, which is the cornerstone of high concurrency. In python3.x, support for the process has also been increased through Asyncio and async\await. In the 2.7 environment used by the author, Greenlet can also be used, after which a post will be introduced.

References:

https://www.python.org/dev/peps/pep-0342/
http://www.dabeaz.com/coroutines/
Https://en.wikipedia.org/wiki/Coroutine#Implementations_for_Python

Python Enhanced Generator-coroutine