Go deep into the iterator and generator in Python

This article mainly introduces the iterator and generator in Python, which involves many important features in Python. For more information, see in Python, many objects can be directly traversed through the for statement, such as list, string, dict, etc. These objects can be called iteratable objects. As to which objects can be accessed by iteration, you need to know the knowledge about the iterator.

Iterator

The iterator object must support the iterator protocol. In Python, The iterator protocol is used to implement the _ iter _ () and next () Methods of the object. The _ iter _ () method returns the iterator object itself; the next () method returns the next element of the container, causing a StopIteration exception at the end.

_ Iter _ () and next () Methods

These two methods are the most basic method of the iterator. One is used to obtain the iterator object and the other is used to obtain the next element in the container.

For iteratable objects, you can use the built-in function iter () to obtain its iterator objects:

In this example, the list iterator object is obtained through the iter () method, and then the elements in the list can be accessed through the next () method. When no element is accessible in the container, the next () method throws a StopIteration exception to terminate the iterator.

In fact, when we use the for statement, the for statement automatically obtains the iterator object through the _ iter _ () method, and uses next () method to obtain the next element.

Custom iterator

After learning about the iterator protocol, you can customize the iterator.

In the following example, A MyRange type is implemented. The _ iter _ () method is used to return the object itself as the iterator object. At the same time, the next () method is implemented to obtain the next element in the container. If no element is accessible, a StopIteration exception is thrown.

class MyRange(object): def __init__(self, n):  self.idx = 0  self.n = n def __iter__(self):  return self def next(self):  if self.idx < self.n:   val = self.idx   self.idx += 1   return val  else:   raise StopIteration()class MyRange(object): def __init__(self, n):  self.idx = 0  self.n = n  def __iter__(self):  return self  def next(self):  if self.idx < self.n:   val = self.idx   self.idx += 1   return val  else:   raise StopIteration()

This custom type is similar to the built-in function xrange. Let's take a look at the running result:

myRange = MyRange(3)for i in myRange: print i

Iterator and iteratable object

In the above example, the object myRange is an iteratable object, and it is also an iterator object.

Look at the code below. For an iteratable object, if it is an iterator object itself, there will be the following problems, and there is no way to support multiple iterations.

To solve the problem above, you can define the iteratable type object and the iterator type object respectively, and then the iteratable type object's _ iter __() method to obtain an iterator type object. See the following implementation:

class Zrange: def __init__(self, n):  self.n = n def __iter__(self):  return ZrangeIterator(self.n)class ZrangeIterator: def __init__(self, n):  self.i = 0  self.n = n def __iter__(self):  return self def next(self):  if self.i < self.n:   i = self.i   self.i += 1   return i  else:   raise StopIteration() zrange = Zrange(3)print zrange is iter(zrange)   print [i for i in zrange]print [i for i in zrange]

The code runs as follows:

In fact, the following code shows that the list type is also in the above way. list itself is an iteratable object, and the list iterator object can be obtained through the iter () method:

Generator

In Python, the generator can easily support the iterator protocol. The generator is generated by using the generator function. The generator function can be defined by conventional def statements, but a result is returned at a time using yield instead of returning, automatically implement iterative protocols by suspending and continuing the statuses of each result.

That is to say, yield is a syntactic sugar. The internal implementation supports the iterator protocol, and yield is a state machine that maintains the pending and continuing states.

Let's take a look at the usage of the generator:

In this example, a generator function is defined. The function returns a generator object and can be accessed through iteration through the for statement.

In fact, the generator function returns the iterator of the generator. The term "generator iterator" is generally referred to as "generator ". Note that the generator is a special iterator. As an iterator, the generator must define some methods, one of which is next (). Like the iterator, we can use the next () function to obtain the next value.

Generator Execution Process

Next, let's take a closer look at how the generator works.

From the above example, we can see that the generator function is very different from the common function.

In combination with the above example, we add some print information to further look at the execution process of the generator:

The result shows that:

When the generator function is called, the function only returns a generator object and is not executed.
When the next () method is called for the first time, the generator function is executed and the execution stops at the yield statement.
The Return value of the next () method is the parameter (yielded value) at the yield statement)
When you continue to call the next () method, the function continues to execute the yield statement that was last stopped and stops at the next yield. If yield is not followed, a StopIteration exception is thrown.
Generator expression

Before introducing the generator expression, let's take a look at the List parsing (List comprehensions) that we are familiar with. List Parsing is generally in the following form.

[expr for iter_var in iterable if cond_expr]

Iterate all content in iterable. After each iteration, put the content in iterable that meets the cond_expr condition into iter_var, and then the content in the expression expr should be iter_var, finally, a list is generated using the calculated value of the expression.

For example, generate a list to protect the odd numbers within 50:

[i for i in range(50) if i%2]

Generator expressions are introduced in python2.4. When the sequence is too long and you only need to get one element at a time, you should consider using generator expressions instead of list parsing. The syntax of the generator expression is the same as that of list parsing, except that the generator expression is enclosed by () instead of [], as follows:

(expr for iter_var in iterable if cond_expr)

Let's look at an example:

The generator expression does not create a list, but returns a generator. This generator calculates an entry and generates the entry (yield. The generator expression uses lazy evaluation, which is assigned a value only during retrieval. Therefore, it is more effective to use memory when the list is relatively long.

Let's look at another example:

In this example, we can see that the generator generated by the generator expression itself is an iteratable object and also the iterator itself.

Recursive Generator

The generator can be recursively used like a function. Let's look at a simple example to arrange a sequence in full:

def permutations(li): if len(li) == 0:  yield li else:  for i in range(len(li)):   li[0], li[i] = li[i], li[0]   for item in permutations(li[1:]):    yield [li[0]] + itemfor item in permutations(range(3)): print itemdef permutations(li): if len(li) == 0:  yield li else:  for i in range(len(li)):   li[0], li[i] = li[i], li[0]   for item in permutations(li[1:]):    yield [li[0]] + item for item in permutations(range(3)): print item

The send () and close () Methods of the generator

The generator has two important methods: send () and close ().

Send (value ):
As we have learned before, the next () method can restore the generator state and continue execution. In fact, send () is another method to restore the generator except next.

In Python 2.5, the yield statement becomes the yield expression, that is, yield can have a value, and this value is the parameter of the send () method, so send (None) and next () is equivalent. Similarly, the values returned by next () and send () are the parameters (yielded value) in the yield statement)

Note that before calling the send () method to pass in a non-None value, the generator must be suspended; otherwise, an exception will be thrown. That is to say, the next () statement or send (None) statement is used for the first call, because there is no yield statement to receive this value.

Close ():
This method is used to close the generator. If you call next or send again after the generator is closed, a StopIteration exception is thrown.

The following describes how to use these two methods:

Summary

This article introduces the Python iterator and generator.

• You can customize the iterator type by implementing the _ iter _ () and next () methods corresponding to the iterator protocol. For iteratable objects, the for statement can obtain the iterator through the iter () method and obtain the next element of the container through the next () method.
• Objects of the sequence type, such as the list, can be iterated and the iterator objects exist independently of each other. In the iteration process, each iterator is independent of each other. However, some iteratable objects are iterative objects, so they cannot be used independently.
• The itertools module provides a series of iterators that help you easily use arrays, combinations, cartesian products, or other composite structures.
• A generator is a special iterator that supports the generator protocol internally and does not need to explicitly define the _ iter _ () and next () methods.
• The generator is generated by using the generator function. The generator function can be defined by conventional def statements, but a result is returned at a time using yield instead of returning.