[Reprint] Fully understand Python iteration objects, iterators, generators

Address: Liuzhijun

In understanding the data structure of Python, the container (container), the Iteration object (iterable), the iterator (iterator), the generator (generator), the list/collection/dictionary derivation (list,set,dict Comprehension) Many concepts are mixed together, it is inevitable for beginners to confused, I will use an article to try to put these concepts and their relationship between the clear.

Container (Container)

A container is a data structure that organizes multiple elements together, and the elements in a container can be iterated over and over, and keywords can be used in to not in determine whether an element is contained in a container. Usually such data structures store all the elements in memory (there are some exceptions, not all elements are placed in memory, such as iterators and generator objects) in Python, common container objects are:

• List, deque, ....
• Set, Frozensets, ....
• Dict, Defaultdict, Ordereddict, Counter, ....
• Tuple, Namedtuple, ...
• Str

The container is easier to understand because you can think of it as a box, a house, a cupboard, which can be stuffed with anything. Technically, when it can be used to ask if an element is contained in it, the object can be thought of as a container, such as a list,set,tuples is a container object:

>>>assert1inch[1, 2, 3]#lists>>>assert4 not inch[1, 2, 3]>>>assert1inch{1, 2, 3}#sets>>>assert4 not inch{1, 2, 3}>>>assert1inch(1, 2, 3)#tuples>>>assert4 not inch(1, 2, 3)

Ask if an element is in dict with the Dict key:

' Foo ' ' Bar ' ' Qux ' }assert in dassert'foo'  not  in D  #  ' foo ' is not an element in Dict

Ask if a substring is in string:

 >>> s =  " foobar  "  >>> assert   b   "  In   s  >>> assert   " x  "  not  in   s  >> > assert   " foo   " in  s 

Although the vast majority of containers provide some way to get each of these elements, this is not the ability provided by the container itself, but the ability to iterate over objects to the container, not all of which are iterative, for example: Bloom filter, although Bloom Filter can be used to detect whether an element is contained in a container, but it is not possible to get each of these values in a single device, because Bloom Filter does not store the element in the container at all, but instead maps it to a value in the array by a hash function.

An iterative object (iterable)

As I said earlier, many containers are iterative objects, and there are more objects that are also iterative objects, such as Files,sockets in the open state, and so on. But the object that can return an iterator can be called an iterative object, which may sound a bit confusing, but let's look at an example:

 >>> x = [1, 2, 3]  >>> y = ITER ( x)  >>> z = iter (x)  >>> Next (y)  1>>> next (y)  2>>>  next (z)  1>>> type (x)  <class   " list   " >>>> type (y)  << Span style= "COLOR: #0000ff" >class   " list_ Iterator   "; 

Here x is an iterative object, can iterate objects and containers is a popular term, do not refer to a specific data type, list is an iterative object, Dict is an iterative object, set is also an iterative object. yand z is two independent iterators that hold a state inside the iterator that records the position of the current iteration to facilitate the next iteration to get the correct element. Iterators have a specific type of iterator, for example list_iterator , set_iterator . An iterative object implements the __iter__ method that returns an Iterator object.

When running code:

x = [1, 2, 3] for in x:    ...

The actual performance is:

To decompile the code, you can see that the interpreter is calling the GET_ITER instruction, which is the equivalent of calling iter(x) , and the FOR_ITER instruction is to call the next() method and constantly get the next element in the iterator, but you can't read it directly from the command because he is optimized by the interpreter.

 >>> import   dis  >>> x = [1, 2, 3 >>> Dis.dis ( " for _ in X:pass   " ) 
     1 0 Setup_loop (to 17)  3 Load_name 0 (x)  6 Get_iter  >&    Gt 7 For_iter 6 (to 16 store_name 1 (_)  Jump_absolute 7 >> 16 Pop_block  >> 17 Load_const 0 (None) 20 Return_value 

Iterators (iterator)

So what about iterators? It is a stateful object that can return the next value in the next() container when you invoke the method, and any object that implements the __iter__ and __next__() (implemented in Python2 next() ) method is an iterator that __iter__ returns the iterator itself, __next__ returning the next value in the container , it is not important to throw stopiteration exceptions if there are no more elements in the container, as to how they are implemented.

So, the iterator is the object that implements the factory pattern, and it returns you every time you ask for the next value. There are many examples of iterators, such as itertools what the function returns are iterator objects.

Generate an infinite sequence:

 from Import count>>> counter = count (start=13)>>> Next (counter)13>>> Next (counter)14

To generate an infinite sequence from a finite sequence:

>>> fromItertoolsImportCycle>>> colors = Cycle (['Red',' White','Blue'])>>>Next (colors)'Red'>>>Next (colors)' White'>>>Next (colors)'Blue'>>>Next (colors)'Red'

To generate a finite sequence from an infinite sequence:

>>> fromItertoolsImportIslice>>> colors = Cycle (['Red',' White','Blue'])#Infinite>>> Limited = islice (colors, 0, 4)#finite>>> forXinchLimited: ...Print(x) redwhitebluered

To get a more intuitive sense of the execution process inside an iterator, we customize an iterator that takes the Fibonacci sequence as an example:

classFib:def __init__(self): Self.prev=0 Self.curr= 1def __iter__(self):return Selfdef __next__(self): value=Self.curr Self.curr+=Self.prev Self.prev=valuereturnvalue>>> f =Fib ()>>> list (islice (f, 0, 10))[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

The FIB is both an iterative object (because it implements the __iter__ method) and an iterator (because it implements the __next__ method). Instance variables prev and curr the state within the user's maintenance iterator. next()do two things each time the method is called:

1. Modify state for Next call next() method
2. Generates a return result for the current call

An iterator is like a lazy-loaded factory that returns a value when someone needs it, and waits for the next call when it is not called.

Generator (Generator)

The builder is one of the most attractive features of the Python language, and the generator is a special kind of iterator, but it's more elegant. It doesn't need to be written and done like the class above __iter__() __next__() , just a yiled keyword. The generator must be an iterator (and vice versa), so any generator also generates values in a lazy-loaded pattern. Examples of Fibonacci sequences implemented with generators are:

def fib ():     = 0, 1 while      True:        yield  curr        = curr, Curr + prev >>> f = fib ()>>> list (islice (f, 0, ten)) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

fibis a common Python function, and its special place is that there is no keyword in the function body return , and the return value of the function is a generator object. When execution f=fib() returns a generator object, the code in the body of the function does not execute, and the code is actually executed only when the next is shown or implicitly called.

The builder is a very powerful programming structure in Python that can be used to write streaming code with less intermediate variables, and it can save memory and CPU compared to other container objects, although it can do similar functions with less code. Now it's time to refactor your code and see something like this:

def something ():     = []    for in  ...:        result.append (x)    return result

Can be replaced with a generator function:

def iter_something ():      for inch  ...:         yield x

Generator Expressions (Generator expression)

A builder expression is a list-down version of the builder that looks like a list derivation, but it returns a generator object instead of a list object.

 for  in range (+)>>> a<generator object <genexpr> at 0x401f08>>> > sum (a)285

Summarize

• A container is a collection of elements, and STR, list, set, Dict, file, sockets objects can all be considered containers, and containers can be iterated (used in statements such as For,while), so they are referred to as iterative objects.
• An iterative object implements the __iter__ method that returns an Iterator object.
• The iterator holds an internal state field that records the next iteration return value, implements the __next__ sum method, and the __iter__ iterator does not load all the elements into memory at once, but produces the returned results when needed.
• The generator is a special iterator whose return value is not passed through return but is used yield .

Reference Link: https://docs.python.org/2/library/stdtypes.html#iterator-types

Liaoche the generator and the iterator are pretty good: generators, iterators

Generator generator, save the algorithm, generally for the traversal can get the algorithm all the value, use for traversal can get all the value generator is an iterative object, can iterate objects like List tuple dict.

Within a function as long as it contains yield is a generator

The generator function cannot be called directly, and the function call displays the result directly when you want to call the generator only when the function call shows that this is a generator. So if you want to use this generator (function), you need to be a generator object, and if there is no generator object, the next () function call will always return the yield next () function called the Generator object

The next () function calls the generator object each time it encounters yield, returns a value, and saves the current generator state again when the next () function is called, the generator generates a second value

A generator is always an iterator (one-way) iterator

After the function iter (iterable) processing can iterate the object and become an iterator

An object that can be called by the next () function and continually returns the next value is called an iterator: Iterator.

Generators are iterator objects

The generator looked dizzy and made up again.

[Reprint] Fully understand Python iteration objects, iterators, generators