5.1.24 python list generation, generator, iterator object and iterator

• The concept of grammatical sugars
• List-Generated
• Generator (Generator)
• An iterative object (iterable)
• Iterators (Iterator)
• The relationship between iterable, iterator and generator

First, the concept of grammatical sugars

"Grammatical sugars", literally, should be a grammar. "Sugar", can be understood as simple, concise. In fact, we have realized that without these grammars, which are called "grammatical sugars", we can also achieve the corresponding functions, while "syntactic sugar" allows us to implement these functions more concisely and quickly. It's just that the Python interpreter translates the syntax of these specific formats into the same complex code logic, nothing too advanced.

So far, the syntactic sugars we have used and introduced are:

• if...else Ternary expression: can simplify the branch judgment statement, such as x = Y.lower () if Isinstance (y, str) Else y
• With statement: for file operation, can help us to automatically close the file object, so that the code is concise;
• Adorner: You can add enhanced function to the function without changing the function code and function call mode;

Here are a few more two:

• list generation: used to generate a new list
• Generator: for "lazy" generation of an infinite sequence

Ii. List-Generated

As the name implies, a list generation is an expression that is used to generate a specific syntactic form of a list.

1. Syntax format: basic syntax format

[exp for iter_var in iterable]

Working process:

• Iterate over each element in the iterable;
• Each iteration first assigns the result to the Iter_var, and then obtains a new calculated value through exp;
• Finally, all the calculated values obtained by EXP are returned as a new list.

The equivalent of this process:

L = []for iter_var in iterable:    L.append(exp)

Syntax format with filter function

for iter_var in iterable if_exp]

Working process:

• Iterating through each element of the iterable, each iteration evaluates to the true of the IF_EXP expression and, if true, the next iteration if it is false;
• The iterative results are assigned to Iter_var, and then a new calculated value is obtained through exp;
• Finally, all the calculated values obtained by EXP are returned as a new list.

The equivalent of this process:

L = []for iter_var in iterable:    if_exp:        L.append(exp)

Looping nested syntax formats

for iter_var_A in iterable_A for iter_var_B in iterable_B]

Working process:
Each iteration of an element in the iterable_a iterates over all the elements in the Ierable_b.

The equivalent of this process:

L = []for iter_var_A in iterable_A:    for iter_var_B in iterable_B: L.append(exp)

2. Application Scenarios

In fact, the list generation is also a python "syntax sugar", that is, the list generation should be a python to provide a concise form of the build list, the application of list generation can quickly generate a new list. Its main application scenario is to derive a new list based on an existing iterative object.

3. Usage examples

The requirements for several build lists can be achieved by using "Do not use list generation" and "Use List generation" respectively, and then make a comparison summary.

Example 1: Generate a list of numbers from 3 to 10

# 不使用列表生成式实现list1 = list(range(3, 11))# 使用列表生成式实现list2 = [x for x in range(3, 11)]

Example 2: Generate a list of 2n+1 numbers, n is a number from 3 to 11

# 不使用列表生成式实现list3 = []for n in range(3, 11): list3.append(2*n + 1)# 使用列表生成式实现list4 = [2*n + 1 for n in range(3, 11)]

Example 3: Filter out an element with a value greater than 20 in a specified list of numbers

L = [3， 7， 11， 14，19， 33， 26， 57， 99]# 不使用列表生成式实现list5 = []for x in L: if x < 20: list5.append(x)# 使用列表生成式实现list6 = [x for x in L if x > 20]

Example 4: Calculates the full array of two sets and saves the result as a new list

L1 = [‘香蕉‘, ‘苹果‘, ‘橙子‘]L2 = [‘可乐‘, ‘牛奶‘]# 不使用列表生成式实现list7 = []for x in L1: for y in L2: list7.append((x, y))# 使用列表生成式实现list8 = [(x, y) for x in L1 for y in L2]

Example 5: Converting a dictionary into a list of tuples consisting of a tuple in the format (key, value)

D = {‘Tom‘: 15, ‘Jerry‘: 18, ‘Peter‘: 13}# 不使用列表生成式实现list9 = []for k, v in D.items(): list9.append((k, v))# 使用列表生成式实现list10 = [(k, v) for k, v in D.items()]

As you can see, the use of list generation is really convenient and concise, and using one line of code is done.

4. List Generation vs. higher-order functions such as map (), filter ()

I think you should have already found that the function of the list generated here is similar to the function of the map () and filter () higher-order functions described in the previous article, such as the following two examples:

Example 1: Converting all strings in a list to lowercase, non-string elements are preserved as-is

L = [‘TOM‘, ‘Peter‘, 10, ‘Jerry‘]# 用列表生成式实现list1 = [x.lower() if isinstance(x, str) else x for x in L]# 用map()函数实现list2 = list(map(lambda x: x.lower() if isinstance(x, str) else x, L))

Example 2: Converting all strings in a list to lowercase, non-string element removal

L = [‘TOM‘, ‘Peter‘, 10, ‘Jerry‘]# 用列表生成式实现list3 = [x.lower() for x in L if isinstance(x, str)]# 用map()和filter()函数实现list4 = list(map(lambda x: x.lower(), filter(lambda x: isinstance(x, str), L)))

For most requirements, the use of list generation and the use of higher-order functions can be achieved. However, some high-order functions, such as map () and filter (), return value types in Python3 into ITERAOTR (iterator) objects (the return value type in the Python2 is list), which is obviously more appropriate for an iterative object with a large or infinite number of elements. Because you can avoid unnecessary waste of memory space. The concept of iterators is described separately below.

Third, generator (Generator)

In terms of name, the generator should be used to generate data.

1. The role of the generator

The new data is continuously generated according to an algorithm until the end of a specified condition is satisfied.

2. How the generator is constructed

There are two ways to construct a generator:

• Generated using a similar list-generated form (2*n + 1 for N in range (3, 11))
• Use a function that contains yield to generate

If the calculation process is relatively simple, you can directly change the list generation to generator, but if the computational process is more complex, you can only construct generator through a function that contains yield.

Description: In previous versions of Python 3.3, the iteration function method was not allowed to include the return statement.

3. Builder Construction Instance

# 使用类似列表生成式的方式构造生成器g1 = (2*n + 1 for n in range(3, 6))# 使用包含yield的函数构造生成器def my_range(start, end): for n in range(start, end): yield 2*n + 1g2 = my_range(3, 6)print(type(g1))print(type(g2))

Output Result:

<class ‘generator‘><class ‘generator‘>

4. The execution process of the generator and the execution of the feature generator:

During execution, the yield keyword is interrupted execution, and the next call continues from where it was last interrupted.

Characteristics of the generator:

• The corresponding data is generated only when it is called
• Record the current location only
• Only next, not prev.

5. How generators are called

There are two ways to call the generator to produce a new element:

• Call the built-in next () method
• Using loops to traverse a generator object (recommended)
• Call the Send () method of the Builder object

Example 1: Using the next () method to traverse the generator

print(next(g1))print(next(g1))print(next(g1))print(next(g1))

Output Result:

call last):  File "***/generator.py", line 26, in <module> print(next(g1))StopIteration

print(next(g2))print(next(g2))print(next(g2))print(next(g2))

Output Result:

call last):  File "***/generator.py", line 31, in <module> print(next(g2))StopIteration

As you can see, when you use the next () method to traverse the generator, the last is to throw an StopIeration exception to terminate.

Example 2: Using Loop traversal builder

for x in g1:    print(x)for x in g2: print(x)

The output of the two loops is the same:

7911

As you can see, using loops to traverse the generator is relatively concise and does not throw an exception at the end StopIeration . Therefore, it is recommended to traverse the generator in a circular fashion.

It is necessary to note that if the generator function has a return value, to get the return value, you can only pass through the next () in a while loop, and finally by catching the StopIteration exception

Example 3: Calling the Generator object's Send () method

def my_range(start, end):    for n in range(start, end): ret = yield 2*n + 1 print(ret)g3 = my_range(3, 6)

print(g3.send(None))print(g3.send(‘hello01‘))print(g3.send(‘hello02‘))

Output Result:

7hello019hello0211

print(next(g3))print(next(g3))print(next(g3))

Output Result:

7None9None11

Conclusion:

• Next () calls yield, but does not pass a value to it
• Send () will call yield, and it will pass a value (the value will be the result of the current yield expression)

It is important to note that the first call to the generator's send () method, the parameter can only be none, otherwise an exception will be thrown. It is also possible to call the next () method before calling the Send () method, in order for the generator to enter the yield expression first.

6. Generator vs. List generation

Now that you can create a new list directly from a list build, why do you have a generator?

Because the list generation is to create a new list directly, it will store all of the data in memory at once, and there are a few problems with the following:

• Limited memory capacity, so the list capacity is limited;
• When the amount of data in the list is very large, it takes up a lot of memory space, if we just need to access the previous finite elements, it will cause a great waste of memory resources;
• When the amount of data is large, the return time of the list generation will be very slow;

The elements in the generator are calculated according to the specified algorithm, and the corresponding data is generated only when the call is made. This eliminates the need to generate all the data at once, thus saving a lot of memory space, which makes the number of elements generated is almost unlimited, and the return time of the operation is very fast (just create a variable).

We can do an experiment: compare the 10 million-digit list to see when the results are returned and the amount of memory space used in the list generation and generator:

Import timeimport Systime_start = Time.time () g1 = [x for x in Range (10000000)]time_end = Time.time () print ( ' list-generated return result takes up memory size:%s '% sys.getsizeof ( G1)) def my_rangefor x in range (start, end): yield Xtime_start = Time.time () g2 = My_Range (0, 10000000) Time_end = Time.time () print ( ' generator returns results time spent:%s '% _end-time_start)) Print ( "generator returns results in memory size:%s '% sys.getsizeof (G2))    

Output Result:

列表生成式返回结果花费的时间： 0.8215489387512207列表生成式返回结果占用内存大小：81528056生成器返回结果花费的时间： 0.0生成器返回结果占用内存大小：88

As a result, the generator returns almost 0 of the time, resulting in a much smaller amount of memory space than the list generator.

Four, an iterative object (iterable)

We often see "iterable" in Python's documentation, which means "iterative objects". So what is an iterative object?
An object that can be used directly for a for loop is called an iterative object (iterable).

The data types that we already know are available as iterations (available for loop) are:

• Collection data types: such as list, tuple, dict, set, str, etc.
• Generator (Generator)

You can use Isinstance () to determine whether an object is a Iterable object:

import Iterableprint(isinstance([], Iterable))

V. Iterators (Iterator) 1. Definition of iterators

an object that can be called by the next () function and continually returns the next value is called an iterator: Iterator.

It is obvious that the generator mentioned above is also an iterator. Of course, we can use Isinstance () to verify:

from collections import Iteratorprint(isinstance((x for x in range(5)), Iterator))

The output is:True

2. Understanding of Iterators

In fact, the iterator object in Python represents a data stream, and iterator can be continuously returned to the next data by the next () function call until no data can be returned with an StopIteration exception. This data stream can be viewed as an ordered sequence, but we cannot know the length of the sequence in advance. At the same time, the calculation of the iterator is inert, and the next data is calculated and returned only when the next () function is passed. (This section is from here)

The generator is also the case, because the generator is also an iterator.

Vi. the relationship between iterable, iterator and generator

• The Builder object is both an iterative object and an iterator: we already know that the generator can function not only with a for loop, but also by the next () function, which is constantly called and returns the next value until the last throw stopiteration error means that the next value cannot continue to be returned. In other words, the generator satisfies the definition of an iterative object and an iterator;
• The iterator object must be an iterative object, and vice versa: For example, the collection data types such as list, dict, and Str are iterative objects, but not iterators, but they can generate an iterator object from the ITER () function.

That is, iterators, generators, and iterative objects can all be iterated with a for loop, and generators and iterators can also be called by the next () party function and return the next value.

5.1.24 python list generation, generator, iterator object and iterator