Fully understand Python iteration objects, iterators, generators

In understanding the data structure of Python, the container (container), the Iteration object (iterable), the iterator (iterator), the generator (generator), the list/collection/dictionary derivation (list,set,dict Comprehension) Many concepts are mixed together, it is inevitable for beginners to confused, I will use an article to try to put these concepts and their relationship between the clear.

Container (Container)

A container is a data structure that organizes multiple elements together, and the elements in a container can be iterated over one after another, using in, the NOT keyword, to determine whether an element is contained in a container. Usually such data structures store all the elements in memory (there are some exceptions, not all elements are placed in memory, such as iterators and generator objects) in Python, common container objects are:

List, deque, .... Set, Frozensets, ..... dict, Defaultdict, Ordereddict, Counter, ..... tuple, namedtuple, ... str

The container is easier to understand because you can think of it as a box, a house, a cupboard, which can be stuffed with anything. Technically, when it can be used to ask if an element is contained in it, the object can be thought of as a container, such as a list,set,tuples is a container object:

>>> assert 1 in [1, 2, 3] # lists >>> assert 4 not in [1, 2, 3] >>> assert 1 in {1, 2, 3 # sets >>> assert 4 not in {1, 2, 3} >>> assert 1 in (1, 2, 3) # tuples >>> Asse RT 4 Not in (1, 2, 3)

Ask if an element is in dict with the Dict key:

>>> d = {1: ' foo ', 2: ' Bar ', 3: ' Qux '} >>> assert 1 in D >>> assert ' foo ' isn't in D # ' Foo ' is not Elements in the Dict

Ask if a substring is in string:

>>> s = ' foobar ' >>> assert ' B ' in S >>> assert ' X ' isn't in S >>> assert ' foo ' in S

Although the vast majority of containers provide some way to get each of these elements, this is not the ability provided by the container itself, but the ability to iterate over objects to the container, not all of which are iterative, for example: Bloom filter, although Bloom Filter can be used to detect whether an element is contained in a container, but it is not possible to get each of these values in a single device, because Bloom Filter does not store the element in the container at all, but instead maps it to a value in the array by a hash function.

An iterative object (iterable)

As I said earlier, many containers are iterative objects, and there are more objects that are also iterative objects, such as Files,sockets in the open state, and so on. But the object that can return an iterator can be called an iterative object, which may sound a bit confusing, but let's look at an example:

>>> x = [1, 2, 3] >>> y = iter (x) >>> z = iter (x) >>> Next (Y) 1 >>> Next (y) 2 >>> next (z) 1 >>> type (x) <class ' list ' > >>> type (y) <class ' List_iterato R ' >

Here x is an iterative object, iterative objects and containers is a popular term, does not mean a specific data type, list is an iterative object, Dict is an iterative object, set is also an iterative object. Y and Z are two independent iterators that hold a state inside the iterator that records where the current iteration is located, so that the correct element is available for the next iteration. Iterators have a specific type of iterator, such as List_iterator,set_iterator. An iterative object implements the __iter__ method, which returns an Iterator object.

When running code:

x = [1, 2, 3] for elem in x: ...

The actual performance is:

Decompile The snippet, and you can see that the interpreter is calling the Get_iter directive, which is equivalent to calling ITER (x), and the for_iter instruction is invoking the next () method to constantly get the next element in the iterator, but you can't just read it from the command. Because he was optimized by the interpreter.

>>> import dis  >>> x = [1, 2, 3]   >>> dis.dis (' For _ in x: pass ')     1            0 SETUP_LOOP               14  (to 17)                  3 LOAD_NAME                 0  (x)                  6 GET_ITER           >>    7 FOR_ITER                  6  (to 16)                 10 store_name                1  (_)                 13 JUMP_ABSOLUTE             7          >>   16 pop _block          >>   17 load_const                0  (None)                 20 RETURN_VALUE   

Iterators (iterator)

What about iterators? It is a stateful object that can return the next value in the container when you call the next () method, and any object that implements the __iter__ and __next__ () (Implementation of the Next ()) method in Python2 is an iterator, __iter__ Returns the iterator itself, __NEXT__ returns the next value in the container, and throws a Stopiteration exception if there are no more elements in the container, which is not important as to how they are implemented.

So, the iterator is the object that implements the factory pattern, and it returns you every time you ask for the next value. There are many examples of iterators, such as the Itertools function that returns an Iterator object.

Generate an infinite sequence:

>>> from Itertools import count >>>wmyl88.com counter = count (start=13) >>> Next (counter) 13 >>> Next (counter) 14

To generate an infinite sequence from a finite sequence:

>>> from itertools import cycle >>> colors = cycle ([' Red ', ' white ', ' Blue ') >>> Next (colors) ' Red ' >>> Next (colors) ' White ' >>> Next (colors) ' Blue ' >>> Next (colors) ' red '

To generate a finite sequence from an infinite sequence:

>>> from itertools import islice >>> colors = cycle ([' Red ', ' white ', ' Blue ']) # infinite >>> Limited = islice (colors, 0, 4) # finite >>> for x in limited: ... print (x) Red white blue Red

To get a more intuitive sense of the execution process inside an iterator, we customize an iterator that takes the Fibonacci sequence as an example:

Class fib:      def __init__ (self):           self.prev = 0           self.curr = 1         def __iter__ (self):           return self          def __next__ (self):           value = self.curr          self.curr +=  self.prev          self.prev = value           return value     >>>  f = fib ()   >>> list (Islice (f, 0, 10))   [1, 1,  2, 3, 5, 8, 13, 21, 34, 55]  

The FIB is both an iterative object (because it implements the __iter__ method) and an iterator (because the __next__ method is implemented). Instance variables prev and Curr users maintain the state inside the iterator. Do two things each time you call the next () method:

The next call to the next () method modifies the state to generate a return result for the current call

An iterator is like a lazy-loaded factory that returns a value when someone needs it, and waits for the next call when it is not called.

Generator (Generator)

The builder is one of the most attractive features of the Python language, and the generator is a special kind of iterator, but it's more elegant. It does not need to write the __iter__ () and __next__ () methods as in the class above, only one yiled keyword is required. The generator must be an iterator (and vice versa), so any generator also generates values in a lazy-loaded pattern. Examples of Fibonacci sequences implemented with generators are:

def fib (): prev, Curr = 0, 1 while True:yield curr prev, Curr = Curr, Curr + prev >> ;> f = fib () >>> list (Islice (f, 0, 10)) [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

The fib is a normal Python function, and its special place is that there is no return keyword in the function body, and the return value of the function is a generator object. When execution F=fib () Returns a generator object, the code in the body of the function does not execute, and the code is actually executed only when the next is shown or implicitly called.

The builder is a very powerful programming structure in Python that can be used to write streaming code with less intermediate variables, and it can save memory and CPU compared to other container objects, although it can do similar functions with less code. Now it's time to refactor your code and see something like this:

def something (): result = [] for ...: result.append (x) return result

Can be replaced with a generator function:

Def iter_something (): For ... in ...: Yield x

Generator Expressions (Generator expression)

A builder expression is a list-down version of the builder that looks like a list derivation, but it returns a generator object instead of a list object.

>>> a = (x*x for x in range) >>> a <generator object <genexpr> at 0x401f08> >>& Gt Sum (a) 285

Summarize

A container is a collection of elements, and STR, list, set, Dict, file, sockets objects can all be considered containers, and containers can be iterated (used in statements such as For,while), so they are referred to as iterative objects. An iterative object implements the __iter__ method, which returns an Iterator object. The iterator holds an internal state field that records the next iteration return value, which implements the __next__ and __iter__ methods, and the iterator does not load all the elements into memory at once, but produces the returned results when needed. A generator is a special iterator whose return value is not by return, but by yield.

Fully understand Python iteration objects, iterators, generators