Python yield Generator detailed

Directory

• Generator Foundation
• Generator applications
  • Generator basic applications
  • Generator Advanced Applications
• Precautions

Body

This article will introduce yield and generator in detail, including the following: What generator, how to generate generator, generator features, generator basic and advanced application scenarios, Precautions for use in generator. This article does not include enhanced generator, or pep342 related content, which is described later in the blog post.

Generator base back to top

In the function definition of Python, as long as a yield expression appears, in fact it is defined as a generator functionthat calls this generator The function return value is a generator. This common function call differs, for example:

Def gen_generator ():    yield 1def gen_value ():    return 1    if __name__ = = ' __main__ ':    ret = Gen_generator () C4/>print ret, type (ret)    #<generator object gen_generator at 0x02645648> <type ' generator ' >    ret = Gen_value ()    print ret, type (ret)    # 1 <type ' int ' >

As can be seen from the above code, the Gen_generator function returns a generator instance, generator has the following special:

• Following the iterator (iterator) protocol, the iterator Protocol needs to implement the __iter__, next interface
• Can pause the execution of code in the body of a function by repeatedly entering and returning multiple times

Here's a look at the test code:

>>> def gen_example ():

... print ' before any yield '

... yield ' first yield '

... print ' between yields '

... yield ' second yield '

... print ' no yield anymore '

...

>>> Gen = Gen_example ()

>>> Gen.next () # First Call next

Before any yield

' First yield '

>>> Gen.next () # Second call to Next

Between yields

' Second yield '

>>> Gen.next () # Third call to Next

No yield anymore

Traceback (most recent):

File "<stdin>", line 1, in <module>

Stopiteratio

Calling the Gen example method does not output anything, stating that the code for the function body has not started executing. When the next method of generator is called, Generator executes to the yield expression, returns the content of the yield expression, and pauses (hangs) in this place, so the first call to next prints the first sentence and returns "primary yield". pausing means that the method's local variables, pointer information, and running environment are saved until the next call to the next method is resumed. After the second call to next, the last yield is paused, and the next () method is called again, and the stopiteration exception is thrown.

Because the For statement can automatically capture the stopiteration exception, the most common way to generator (essentially any iterator) is to use it in a loop:

1 def generator_example (): 2     yield     5 if __name__ = = ' __main__ ': 6 for     E in Generator_example (): 7
   
    print E8         # output 1 2
   

What is the difference between the generator of generator function and the normal function?

(1) function is run from the first line each time, and generator from where the last yield began

(2) A function call returns one (group) value at a time, and generator can return multiple times

(3) function can be repeated many times, and a generator instance cannot continue to invoke after the last value of yield or return.

Using yield in a function and then calling the function is a way to generate generator. Another common way is to use generator Expression,for example:
>>> Gen = (x * x for X in Xrange (5))
>>> Print Gen
<generator Object <genexpr> at 0x02655710>
　　

Generator application back to top

Generator basic applications

Why use generator, the most important reason is that you can generate and "return" results on demand , rather than produce all the return values at once, and sometimes do not know "all return values." For example, the following code

1     range_num = 1002     for i in [x*x to X in RANGE (Range_num)]: # The first method: Iterate over the list 3         # do sth for example4         print i5 6 for     I in (x*x for X in range (Range_num)): # second method: Iterate over generator 7         # do STH for example8         print I

In the above code, the two for statement output is the same, and the code literally appears to be the difference between the brackets and the parentheses. But this difference is very big, the first method return value is a list, the second method returns a Generator object. As Range_num becomes larger, the first method returns a larger list and consumes more memory, but there is no difference between the second method.

Let's look at an example that can "return" an infinite number of times:

def fib ():    A, B = 1, 1 while    True:        yield A        

This generator has the ability to generate countless "return values" that the user can decide when to stop iterating

Generator Advanced Applications

Use Scenario One:

Generator can be used to generate a data stream, generator does not immediately produce a return value, but rather wait until it is needed to produce a return value, which is equivalent to an active pull process, such as now has a log file, each row produces a record, for each record, People in different departments may handle different ways, but we can provide a common, on-demand data flow.

1 def gen_data_from_file (file_name): 2 for line in     file (file_name): 3         yield line 4  5 def gen_words (line): 6
   for Word in (W to W in Line.split () if W.strip ()): 7         yield Word 8  9 def count_words (file_name):     word_map = {}11 in     gen_data_from_file (file_name): ' For word ' in ' gen_words line '         : ' If Word not ' in             word_map:14                 Word_map[word] = 015             Word_map[word] + =     return word_map17 def count_total_chars (file_name): 19     Total = 020 to line in     Gen_data_from_file (file_name): all         + = Len (line)     return total23     if __name__ = = ' __main__ ':     print count_words (' Test.txt '), Count_total_chars (' Test.txt ')

The above example comes from a 08-year Pycon lecture. Gen_words Gen_data_from_file is a data producer, and Count_words Count_total_chars is the consumer of data. You can see that data is pulled only when it is needed, rather than being prepared in advance . In addition Gen_words (W for W in Line.split () if W.strip ()) also produces a generator

Use Scenario Two:

In some programming scenarios, one thing may require some logic to be executed, and then wait for a period of time, or wait for an asynchronous result, or wait for a state before proceeding to another part of the logic. For example, in a microservices architecture, service A executes a logic, goes to service B to request some data, and then resumes execution on service A. Or in game programming, a skill is divided into multiple segments, performing part of the action (effect), then waiting for a period of time before continuing. We generally use the callback (callback) approach for situations that require waiting and do not want to be blocked. Here's a simple example:

1 def Do (a): 2     print ' Do ', A3     Callbackmgr.callback (5, lambda a = A:post_do (a)) 4 5 def POST_DO (a): 6     print ' Post _do ', a

Here the Callbackmgr registered a 5s time, 5s after the call to the lambda function, it can be seen that a logic is split into two functions, but also need to pass the context (as the parameter a here). We use yield to modify this example, and the yield return value represents the time of the wait.

1 @yield_dec2 def do (a): 3     print ' do ', A4     yield     print ' Post_do ', a

Here you need to implement a yieldmanager, through Yield_dec this decrator will do this generator register to Yieldmanager, and after 5s call next method. The yield version implements the same functionality as callbacks, but looks much clearer. Here is a simple implementation for reference:

#-*-Coding:utf-8-*-import sys# import timerimport typesimport timeclass Yieldmanager (object): Def __init__ (self, tic K_delta = 0.01): Self.generator_dict = {} # Self._tick_timer = Timer.addrepeattimer (Tick_delta, lambda:self . Tick ()) def tick (self): cur = time.time () for gene, T in Self.generator_dict.items (): If cur            >= T:self._do_resume_genetator (gene,cur) def _do_resume_genetator (Self,gene, cur): try: Self.on_generator_excute (gene, cur) except stopiteration,e:self.remove_generator (gene) EX Cept Exception, e:print ' unexcepet error ', type (e) self.remove_generator (gene) def Add_generator (Self, Gen., Deadline): Self.generator_dict[gen] = Deadline def remove_generator (self, Gene): Del Self.gen  Erator_dict[gene] def on_generator_excute (self, gen, cur_time = None): t = gen.next () Cur_time = Cur_time or Time.time () sElf.add_generator (Gen, T + cur_time) G_yield_mgr = Yieldmanager () def yield_dec (func): Def _inner_func (*args, **kwargs): Gen = func (*args, **kwargs) if Type (gen) is types.     GeneratorType:g_yield_mgr.on_generator_excute (Gen) return Gen return _inner_func@yield_decdef Do (a): print ' do ', a yield 2.5 print ' post_do ', a yield 3 print ' Post_do again ', aif __name__ = = ' __main__ ': D O (1) for I in range (1, ten): print ' Simulate a timer,%s seconds passed '% i time.sleep (1) g_yield_ Mgr.tick ()

Note: Back to top

(1) yield is not nested!

1 def visit (data): 2 for     elem in Data:3         if Isinstance (elem, tuple) or isinstance (Elem, List): 4             Visit (elem) #  Here value retuened is generator 5         else:6             yield elem 7              8 if __name__ = = ' __main__ ': 9     for E in visit ([1, 2, (3, 4), 5]): Ten         print E

The code above accesses every element in the nested sequence, and we expect the output to be 1 2 3 4 5, and the actual output is 1 2 5. Why, as the note shows, visit is a generator function, so the 4th line returns the Generator object, and the code does not iterate over the generator instance. Then change the code and iterate over the temporary generator.

def visit (data): For    elem in data:        if Isinstance (elem, tuple) or isinstance (Elem, list): For            E in visit (elem) :                yield e        else:            yield elem

Or you can use yield from in python3.3, which is added in pep380

1 def visit (data): 2 for     elem in Data:3         if Isinstance (elem, tuple) or isinstance (Elem, List): 4             yield from visit (elem) 5         else:6             yield elem

(2) Use return in generator function

In the Python doc, it is explicitly mentioned that you can use return and throw the stopiteration exception when generator executes here.

1 def gen_with_return (range_num): 2     if range_num < 0:3         return 4     else:5 for         i in Xrange (range_num): 6< C10/>yield I 7  8 if __name__ = = ' __main__ ': 9     Print List (Gen_with_return ( -1))     Print List (gen_with_ Return (1))

However, return in the generator function is not allowed with any return value.

1 def gen_with_return (range_num): 2     if range_num < 0:3         return     else:5 for         i in Xrange (range_num ): 6             yield I

The above code will error:syntaxerror: ' Return ' with argument inside generator

References:

http://www.dabeaz.com/generators-uk/
https://www.python.org/dev/peps/pep-0380/
Http://stackoverflow.com/questions/231767/what-does-the-yield-keyword-do
Http://stackoverflow.com/questions/15809296/python-syntaxerror-return-with-argument-inside-generator

Python yield Generator detailed