In-depth explanation of iterators and generators in Python

In Python, many objects can be traversed directly through a for statement, such as list, String, Dict, and so on, which can be called an iterative object. As for which objects can be accessed iteratively, it is necessary to understand the knowledge of iterators.

Iterators

The iterator object requires an object that supports the iterator protocol, in Python, the __iter__ () and Next () methods that implement the object are the support iterator protocol. where the __iter__ () method returns the Iterator object itself; the next () method returns the next element of the container and throws an Stopiteration exception at the end.

__iter__ () and Next () methods

These two methods are the most basic method of iterators, one to get the iterator object, and one to get the next element in the container.

For an iterative object, you can use the built-in function iter () to get its iterator object:

example, the list iterator object is obtained through the ITER () method, and then the elements in the list can be accessed through the next () method. When there are no accessible elements in the container, the next () method throws an Stopiteration abort iterator.

In fact, when we use the For statement, the For statement automatically obtains the iterator object through the __iter__ () method, and gets the next element through the next () method.

Custom iterators

Once you understand the iterator protocol, you can customize the iterator.

The following example implements a myrange type, which implements the __iter__ () method, which returns the object itself as an iterator object, and implements the next () method to get the next element in the container, when there is no accessible element. Throws a Stopiteration exception.

Class MyRange (object): Def __init__ (self, N):  self.idx = 0  SELF.N = n def __iter__ (self):  return self def Next (self):  if Self.idx < self.n:   val = self.idx   Self.idx + = 1   return val  else:   raise Stopiteration () class MyRange (object): Def __init__ (self, N):  self.idx = 0  SELF.N = n  def __iter__ (self): C12/>return Self  def next (self):  if Self.idx < self.n:   val = self.idx   Self.idx + = 1   return Val  else:   raise Stopiteration ()

This custom type is similar to the built-in function xrange, and looks at the results of the operation:

MyRange = MyRange (3) for I in Myrange:print I

Iterators and Iterative objects

In the example above, the MyRange object is an iterative object, and it is itself an iterator object.

Look at the following code, for an iterative object, if it is itself an iterator object, there will be a problem, there is no way to support multiple iterations.

To solve the above problem, you can define an object that iterates type objects and iterator types, and then you can iterate over the __iter__ () method of the type object to get an object of an iterator type. Look at the following implementations:

Class Zrange:def __init__ (self, N):  SELF.N = n def __iter__ (self):  return Zrangeiterator (SELF.N) class Zrangeiterator:def __init__ (self, N):  self.i = 0  SELF.N = n def __iter__ (self):  return self def Next: 
  if self.i < self.n:   i = self.i   self.i + = 1   return i  else:   raise stopiteration () Zrange = Zrange (3) Print Zrange is ITER (zrange)   Print [i-I in zrange]print [i-I in Zrange]

The code runs with the following results:

In fact, the following code can be seen, the list type is also in the way above, the list itself is an iterative object, through the ITER () method can get the list iterator object:

Generator

In Python, the use of generators makes it easy to support iterator protocols. Generators are generated through the generator function, which can be defined by a regular DEF statement, but instead of returning it, it returns a result with yield once, suspending and continuing their state between each result, to automatically implement the iterative protocol.

In other words, yield is a syntactic sugar, and the internal implementation supports the iterator protocol, while the yield is a state machine that maintains the status of suspend and continue.

Here's how the generator is used:

In this example, a generator function is defined, and the function returns a generator object, which can then be iterated through the for statement.

In fact, the generator function returns an iterator to the generator. The term "generator iterator" is often referred to as the "generator". It is important to note that the generator is a special kind of iterator. As an iterator, the generator must define some methods, one of which is next (). As with iterators, we can use the next () function to get the next value.

Generator execution Process

Here's a closer look at how the generator works.

As you can see from the example above, generator functions are very different from normal functions.

In conjunction with the above example, we add some printing information and take a closer look at the generator's execution flow:

From the results you can see:

When the generator function is called, the function simply returns a generator object and does not execute.
When the next () method is called for the first time, the generator function begins execution, and execution stops at the yield statement
The return value of the next () method is the argument at the yield statement (yielded value)
When the next () method continues to be called, the function continues at the last stop yield statement and stops at the next yield and throws a stopiteration exception if no yield is followed.
Builder expression

Before we begin to introduce the builder expression, let's look at our familiar list comprehensions, which is usually the following form of list parsing.

[Expr for Iter_var in iterable if COND_EXPR]

Iterate over all the contents of the iterable, after each iteration, put the contents of the iterable satisfies the cond_expr condition into the Iter_var, then the content that should be iter_var in the expression expr, and finally generate a list with the calculated value of the expression.

For example, generate a list to protect within 50 so odd:

[I for I in range () if i%2]

The builder expression is introduced in python2.4, and you should consider using a generator expression instead of list parsing when the sequence is too long and you need to get only one element at a time. The syntax of a generator expression is the same as a list resolution, except that the generator expression is surrounded by (), not [], as follows:

(Expr for Iter_var in iterable if cond_expr)

See an example:

Instead of creating a list, the generator expression returns a generator that "yields" the entry after each calculation of an entry. The generator expression uses lazy evaluation, which is only assigned when retrieved (evaluated), so it is more efficient to use memory when the list is longer.

Continue to see an example:

As you can see from this example, the generator expression produces a builder that itself is an iterative object and is also the iterator itself.

Recursive generators

The generator can be used recursively like a function, and below is a simple example of a sequence that is fully arranged:

def permutations (LI): If Len (li) = = 0:  yield li else: for  i in range (Len (LI)):   li[0], li[i] = Li[i], li[0]
  
   for item in permutations (li[1:]):    yield [li[0]] + itemfor item in permutations (range (3)): Print itemdef permutations ( LI): If Len (li) = = 0:  yield li else:  for I in range (Len (LI)):   li[0], li[i] = Li[i], li[0] for item in   Perm Utations (li[1:]):    yield [li[0]] + item for item in permutations (range (3)): Print item
  

Generator's send () and close () methods

There are two other important methods in the generator: Send () and close ().

Send (value):
From the previous understanding, the next () method can restore the generator State and continue execution, in fact, send () is a method other than next () recovery generator.

In Python 2.5, the yield statement becomes the yield expression, which means that yield can have a value, which is the parameter of the Send () method, so Send (None) and next () are equivalent. Similarly, the return values of next () and send () are the arguments at the yield statement (yielded value)

Note about the Send () method: The generator must be in a pending state before calling send to pass in a value other than none, otherwise an exception will be thrown. That is, the first call is to use the next () statement or send (None), because there is no yield statement to receive this value.

Close ():
This method is used to close the generator, and calling next or send again after the shutdown generator throws a Stopiteration exception.

Here's a look at the use of these two methods:

Summarize

This article describes what Python iterators and generators are about.

• You can customize the iterator type by implementing the __ITER__ () and Next () methods that correspond to the iterator protocol. For an iterative object, the For statement can obtain an iterator through the ITER () method and get the next element of the container through the next () method.
• Objects such as lists of such sequence types, iterative objects and iterator objects exist independently of each other, and each iterator is independent of each other in the iterative process, but the iterators cannot be used independently of the iterator object itself.
• The Itertools module provides a range of iterators that can help users easily use permutations, combinations, Cartesian product, or other composite structures.
• The generator is a special iterator that internally supports the generator protocol and does not need to explicitly define the __iter__ () and Next () methods.
• Generators are generated through generator functions, which can be defined by regular DEF statements, but instead of return, they return one result at a time with yield.

