Python iterative objects, iterators, builder understanding

In understanding Python's data structure, containers (container), Iteration objects (iterable), iterators (iterator), generators (generator), list/set/dictionary derivations (list,set,dict Comprehension) Many concepts are mixed together, it is inevitable for beginners to confused, I will use an article to try to make these concepts and the relationship between them clear.

Relations

Container (Container)
A container is a data structure that organizes multiple elements together, and the elements in a container can be iterated by iteration, and you can use in, the not in keyword to determine whether an element is contained in a container. Typically, this type of data structure stores all the elements in memory (and some of the special columns are not all elements in memory) in Python, the common container objects are:

List, Deque, .....
Set, Frozensets, .....
Dict, Defaultdict, Ordereddict, Counter, ...
Tuple, Namedtuple, ...
Str
The container is easier to understand, because you can think of it as a box, a house, a closet, where you can plug anything. Technically, when it can be used to ask if an element is included in it, the object can be considered a container, such as List,set,tuples, which is a container object:

>>> assert 1 in [1, 2, 3] # lists
>>> assert 4 Not in [1, 2, 3]
>>> assert 1 in {1, 2, 3} # Sets
>>> assert 4 Not in {1, 2, 3}
>>> assert 1 in (1, 2, 3) # tuples
>>> assert 4 Not in (1, 2, 3)

Ask if an element is using the Dict key in Dict:

>>> d = {1: ' foo ', 2: ' Bar ', 3: ' Qux '}
>>> assert 1 in D
>>> assert ' foo ' isn't in D # ' Foo ' is not an element in Dict

Ask if a substring is in string:

>>> s = ' foobar '
>>> assert ' B ' in S
>>> assert ' X ' not in S
>>> assert ' foo ' in S
Although the vast majority of containers provide some way to get each of these elements, this is not the capacity provided by the container itself, but the ability of an iterative object to give the container this capability, and of course not all containers are iterative, for example: Bloom filter, although bloom Filter can be used to detect whether an element is contained in a container, but it is not able to obtain each of the values in the device, because Bloom Filter does not store the elements in the container, but rather by mapping a hash function into a value stored in the array.

Iteration Objects (iterable)
As I said earlier, many containers are iterative objects, and there are more objects that can be iterated, such as files,sockets in the open state, and so on. But any object that can return an iterator can be called an iterative object, which may sound a little confusing, okay, first look at an example:

>>> x = [1, 2, 3]
>>> y = iter (x)
>>> z = iter (x)
>>> Next (Y)
1
>>> Next (Y)
2
>>> Next (z)
1
>>> type (x)
<class ' list ' >
>>> type (y)
<class ' List_iterator ' >

Here x is an iterative object, an iterative object and container is a popular term, not a specific data type, list is an iterative object, Dict is an iterative object, set is also an iterative object. Y and Z are two separate iterators that hold a state inside the iterator that records where the current iteration is located to facilitate getting the correct element at the next iteration. Iterators have a specific type of iterator, such as List_iterator,set_iterator. An iterative object implements the __iter__ and __next__ methods (Python2 is the next method, Python3 is the __next__ method), and the two methods correspond to the built-in functions ITER () and next (). The __iter__ method returns the iterated object itself, which allows him to be both an iterative object and an iterator.

When you run the code:

x = [1, 2, 3]
For Elem in x:
...
The actual implementation is:

Decompile the code, you can see that the interpreter invokes the get_iter instruction, which is equivalent to invoking ITER (x), the for_iter instruction is to call the next () method, and constantly get the next element in the iterator, but you can't read it directly from the command because he has been optimized by the interpreter.

>>> Import Dis
>>> x = [1, 2, 3]
>>> Dis.dis (' for _ in X:pass ')
1 0 setup_loop (to 17)
3 Load_name 0 (x)
6 Get_iter
>> 7 For_iter 6 (to 16)
Ten Store_name 1 (_)
Jump_absolute 7
>> Pop_block
>> load_const 0 (None)
Return_value
Iterator (iterator)
What about the iterator? It is a stateful object that can return the next value in the container when you call the next method, and any object that implements the __next__ () (Python2 Next ()) method is an iterator, and it doesn't matter how it is implemented.

So, the iterator is the object that implements the factory pattern, and it returns to you every time you ask for the next value. There are many examples of iterators, such as the Itertools function that returns an Iterator object.

To generate an infinite sequence:

>>> from Itertools Import count
>>> counter = count (start=13)
>>> Next (counter)
13
>>> Next (counter)
14

To generate an infinite sequence from a finite sequence:

>>> from Itertools Import cycle
>>> colors = cycle ([' Red ', ' white ', ' Blue '])
>>> Next (colors)
' Red '
>>> Next (colors)
' White '
>>> Next (colors)
' Blue '
>>> Next (colors)
' Red '

To generate a finite sequence from an infinite sequence:

>>> from Itertools import islice
>>> colors = cycle ([' Red ', ' white ', ' Blue ']) # infinite
>>> Limited = islice (colors, 0, 4) # finite
>>> for X in limited:
... print (x)
Red
White
Blue
Red

To feel more intuitive about the execution process within the iterator, we customize an iterator, taking the Fibonacci sequence as an example:

Class Fib:
def __init__ (self):
Self.prev = 0
Self.curr = 1

def __iter__ (self):
return self

def __next__ (self):
Value = Self.curr
Self.curr + + Self.prev
Self.prev = value
return value

>>> f = Fib ()
>>> list (islice (f, 0, 10))
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

FIB is both an iterative object (because it implements the __iter__ method) and an iterator (because the __next__ method is implemented). Instance variables prev and Curr users maintain the internal state of the iterator. Do two things each time you call the next () method:

To modify the state for the next call to the next () method
Returns results for the current call generation
An iterator is like a lazy-loading factory, and waits until someone needs it to generate a value to return, and is dormant until the next call.

Builder (Generator)
The generator is one of the most appealing features in the Python language, and the generator is actually a special iterator, but the iterator is more elegant. It does not need to write __iter__ () and __next__ () methods like the class above, but only one yiled keyword is required. The generator must be an iterator (not vice versa), so any generator is also generating values in a lazy load mode. Examples of Fibonacci sequences using generators are:

def fib ():
Prev, Curr = 0, 1
While True:
Yield Curr
Prev, Curr = Curr, Curr + prev

>>> f = fib ()
>>> list (islice (f, 0, 10))
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

FIB is a common Python function, which is the special one where there is no return keyword in the body of the function, and the returned value of the function is a generator object. When the execution F=fib () Returns a generator object, the code in the function body is not executed, and the code is actually executed only when the display or implicitly invokes next.

The builder is a very powerful programming structure in Python that can write streaming code with fewer intermediate variables, and it can save memory and CPU compared to other container objects, but it can achieve similar functionality with less code. Now you can refactor your code and see something like this:

def something ():
result = []
For ... in ...:
Result.append (x)
return result
Can be replaced with a generator function:

Def iter_something ():
For ... in ...:
Yield x

Builder expression (generator expression)
A builder expression is a list-torn builder version that looks like a list derivation, but it returns a generator object instead of a list object.

>>> a = (x*x for x in range (10))
>>> A
<generator Object <genexpr> at 0x401f08>
>>> sum (a)
285