Details about yield and generator in python

This article mainly introduces yield and generator in python from a simple perspective. This article is very detailed and has some reference value for everyone. let's take a look at it. This article mainly introduces yield and generator in python from a simple perspective. This article is very detailed and has some reference value for everyone. let's take a look at it.

Preface

This article introduces yield and generator in detail, including the following content: What generator, generator generation method, characteristics of generator, basic and advanced application scenarios of generator, and precautions for using generator. This article does not include enhanced generator, or pep342, which will be described later.

Generator basics

In the function definition of python, as long as the yield expression (Yield expression) appears, what is actually defined is a generator function that calls thisgenerator functionThe returned value is a generator. This common function call is different, For example:

def gen_generator(): yield 1def gen_value(): return 1 if __name__ == '__main__': ret = gen_generator() print ret, type(ret) #
  
  
    ret = gen_value() print ret, type(ret) # 1 
   
  
 

From the code above, we can see that,gen_generatorThe function returns a generator instance.

Generator has the following special features:

• Following the iterator protocol, the iterator protocol needs to be implemented__iter__ , Next interface

• It can be accessed multiple times and returned multiple times to pause code execution in the function body

The test code is as follows:

>>> Def gen_example ():... print 'before any yield '... yield 'first yield '... print 'between yields '... yield 'second yield '... print 'no yield anymore'...> gen = gen_example () >>> gen. next () # Call nextbefore any yield first yield for the first time> gen. next () # Call nextbetween yields 'second yield '> gen. next () # The third call nextno yield anymoreTraceback (most recent call last): File"
 
  
", Line 1, in
  
   
StopIteratio
  
 

The gen example method is called and no content is output, indicating that the code of the function body has not been executed yet. When the next method of generator is called, generator executes the yield expression, returns the content of the yield expression, and suspends (Suspends) the content in this place, therefore, when you call next for the first time, print the first sentence and return "first yield ". Pause means that the local variables, pointer information, and running environment of the method are saved until the next method is called for recovery. After the second call to next, it will suspend the last yield and call againnext()Will throw the StopIteration exception.

Because the for statement can automatically capture StopIteration exceptions, generator (essentially any iterator) is commonly used in a loop:

def generator_example(): yield 1 yield 2if __name__ == '__main__': for e in generator_example(): print e # output 1 2

What is the difference between a generator generated by a generator function and a common function?

(1) function runs from the first line each time, while generator runs from the last yield.

(2) One (group) value is returned for a function call at a time, while the generator can return multiple

(3) a function can be called multiple times, and a generator instance cannot be called after the last value or return of yield.

Using Yield in a function and then calling this function is a way to generate a generator. Another common method is to usegenerator expression, For example:

　　>>> gen = (x * x for x in xrange(5))　　>>> print gen　　
 
   at 0x02655710>
 

Generator application

Basic application of generator

Why do we use generator? the most important reason is that we can generate and "return" results as needed, instead of generating all the returned values at once. Besides, sometimes we don't know "all the returned values ".

For example, for the following code　

RANGE_NUM = 100 for I in [x * x for x in range (RANGE_NUM)]: # Method 1: iteration of the list # do something for example print I for I in (x * x for x in range (RANGE_NUM): # Method 2: iterate on generator # do something for example print I

In the above code, the two for statement outputs are the same, and the code literally seems to be the difference between Brackets and parentheses. However, the difference is very large. The first method returns a list and the second method returns a generator object. As RANGE_NUM increases, the list returned by the first method is larger, and the memory occupied is larger. However, there is no difference between the second method.

Let's take a look at an example that can return an infinite number of times:

def fib(): a, b = 1, 1 while True: yield a a, b = b, a+b

This generator has the ability to generate numerous "return values". you can decide when to stop iteration.

Generator Advanced Applications

Scenario 1:

Generator can be used to generate data streams. generator does not immediately generate a returned value. Instead, it will generate a returned value when needed. it is equivalent to a pull process. for example, there is a log file, each row generates a record. for each record, different departments may have different processing methods, but we can provide a public, on-demand data stream.

def gen_data_from_file(file_name): for line in file(file_name): yield linedef gen_words(line): for word in (w for w in line.split() if w.strip()): yield worddef count_words(file_name): word_map = {} for line in gen_data_from_file(file_name): for word in gen_words(line):  if word not in word_map:  word_map[word] = 0  word_map[word] += 1 return word_mapdef count_total_chars(file_name): total = 0 for line in gen_data_from_file(file_name): total += len(line) return total if __name__ == '__main__': print count_words('test.txt'), count_total_chars('test.txt')

The above example is from a lecture on PyCon in.gen_words gen_data_from_fileIs the data producer, while count_words count_total_chars is the data consumer. We can see that the data is pulled only when needed, rather than prepared in advance. In addition, gen_words(w for w in line.split() if w.strip()) It also generates a generator

Scenario 2:

In some programming scenarios, one thing may need to execute some logic, then wait for a period of time, or wait for an asynchronous result, or wait for a certain state, and then continue to execute another part of logic. For example, in the microservice architecture, after service A executes A logic, service B requests some data and then continues to execute it on service. Or in game programming, a skill is divided into multiple parts, and a part of the action (effect) is executed first, then wait for a while, and then continue. In this case, we usually use the callback method. The following is a simple example:

 def do(a): print 'do', a CallBackMgr.callback(5, lambda a = a: post_do(a))  def post_do(a): print 'post_do', a

Here, CallBackMgr registers a time after 5s and calls it after 5s.lambdaFunction. it can be seen that a piece of logic is split into two functions, and context transmission is also required (for example, parameter a here ). We use yield to modify this example. the yield return value indicates the waiting time.

 @yield_dec def do(a): print 'do', a yield 5 print 'post_do', a

Here we need to implement a YieldManager, throughyield_decThe decrator registers do generator to YieldManager and calls the next method after 5s. Yield implements the same functions as callback, but it seems much clearer.

A simple implementation is provided below for your reference:

# -*- coding:utf-8 -*-import sys# import Timerimport typesimport timeclass YieldManager(object): def __init__(self, tick_delta = 0.01): self.generator_dict = {} # self._tick_timer = Timer.addRepeatTimer(tick_delta, lambda: self.tick()) def tick(self): cur = time.time() for gene, t in self.generator_dict.items():  if cur >= t:  self._do_resume_genetator(gene,cur) def _do_resume_genetator(self,gene, cur ): try:  self.on_generator_excute(gene, cur) except StopIteration,e:  self.remove_generator(gene) except Exception, e:  print 'unexcepet error', type(e)  self.remove_generator(gene) def add_generator(self, gen, deadline): self.generator_dict[gen] = deadline def remove_generator(self, gene): del self.generator_dict[gene] def on_generator_excute(self, gen, cur_time = None): t = gen.next() cur_time = cur_time or time.time() self.add_generator(gen, t + cur_time)g_yield_mgr = YieldManager()def yield_dec(func): def _inner_func(*args, **kwargs): gen = func(*args, **kwargs) if type(gen) is types.GeneratorType:  g_yield_mgr.on_generator_excute(gen) return gen return _inner_func@yield_decdef do(a): print 'do', a yield 2.5 print 'post_do', a yield 3 print 'post_do again', aif __name__ == '__main__': do(1) for i in range(1, 10): print 'simulate a timer, %s seconds passed' % i time.sleep(1) g_yield_mgr.tick()

Note:

(1) Yield cannot be nested!

def visit(data): for elem in data: if isinstance(elem, tuple) or isinstance(elem, list):  visit(elem) # here value retuened is generator else:  yield elem  if __name__ == '__main__': for e in visit([1, 2, (3, 4), 5]): print e

The above code accesses every element in the nested sequence. the expected output is 1 2 3 4 5, and the actual output is 1 2 5. As shown in the notes, visit isgenerator functionSo the 4th rows returngenerator objectAnd the code does not have this generator instance iteration. Then modify the code and iterate on the temporary generator.

def visit(data): for elem in data: if isinstance(elem, tuple) or isinstance(elem, list):  for e in visit(elem):  yield e else:  yield elem

Or you can useyield fromThis syntax is added to pep380.

 def visit(data): for elem in data:  if isinstance(elem, tuple) or isinstance(elem, list):  yield from visit(elem)  else:  yield elem

(2) use return in generator function

In python doc, return can be explicitly mentioned. when the generator runs here, a StopIteration exception is thrown.

def gen_with_return(range_num): if range_num < 0: return else: for i in xrange(range_num):  yield iif __name__ == '__main__': print list(gen_with_return(-1)) print list(gen_with_return(1))

However,generator functionThe return value in does not contain any returned values.

 def gen_with_return(range_num): if range_num < 0:  return 0 else:  for i in xrange(range_num):  yield i

The above code will report an error:SyntaxError: 'return' with argument inside generator

Summary

The above is a detailed description of yield and generator in python. For more information, see other related articles in the first PHP community!