Python Advanced _ Builder & Builder expression

Directory

• Directory
• Related knowledge points
• Generator
  • The execution process of the builder Fab
  • The difference between a generator and an iterator
  • Benefits of the generator
  • Enhanced generator Features
• Builder expression
  • Example of generator expression model
• Summary

Related knowledge points

Python Advanced _ Iterator & List parsing

Generator

Functions with the yield keyword are called generator (generators) in Python. The Python interpreter treats the function with the yield keyword as a generator. A function or subroutine can only return once, but a generator can pause execution and return an intermediate result- This is the function of the yield statement: Returns an intermediate value to the caller and pauses execution.

EXAMPLE:

in [94]: def Fab(max):...: N, a, B =0,0,1...: whilen < max: ...:yieldB ...: A, B = B, A + b ...: n = n +1...: in [ the]: F = Fab (5) in [ the]: F.next () out[ the]:1in [ the]: F.next () out[ the]:1in [98]: F.next () out[98]:2in [ About]: F.next () out[ About]:3in [ -]: F.next () out[ -]:5in [101]: F.next ()---------------------------------------------------------------------------stopiteration Traceback (most recent) <ipython-input-101-c3e65e5362fb>inch<module> ()---->1F.next () Stopiteration:

Generator fab()The execution process

executing f = fab(5) The statement does not immediately execute fab() the function's code block, but instead returns a Iterable object first!
The fab() code block of the function executes when the FOR Loop statement executes.
When executed to the statement yield b , the fab() function returns an iteration value until the next iteration, the flow of the program goes back to yield b the next statement to continue, and then goes back to the for loop, so the iteration ends. It looks as if a function was yield interrupted several times during normal execution, and each interrupt is yield returned by returning the current iteration value.
As you can see, the generator yield will constantly return the iterator to memory through the keyword, instead of putting the object all in memory at once, thus saving memory space. From this point of view the generator and the iterator are very similar, but if you know more deeply, there is still a difference between the two.

The difference between a generator and an iterator

Another advantage of the generator is that it does not require that you prepare all the elements of the entire iteration in advance, that is, you do not have to store all the elements of the object in memory before you begin to do so. The generator will put the element into memory only when iterating to an element, and before or after that, the element may not exist or be destroyed. This feature makes it particularly well suited for traversing a number of large or infinite class sequence objects, EG. Large File/large set/Large dictionary/Fibonacci sequence etc. This feature is called lazy Evaluation or lazy evaluation (lazy), which can save memory effectively. The lazy evaluation is actually the idea of the cooperative program .

Co -operation: A function call that can be run independently, which can be paused or suspended, and can then be resumed or resumed from where the program flow hangs. When a co-program is suspended, Python is able to get the return value of a property in the middle State from the synergistic program (returned by yield), and when the calling next() method causes the program to flow back into the co-program, it can pass in additional or changed parameters. and continues execution from the last pending statement. This is a method of calling a function similar to a process interrupt. this pending function call and after returning the intermediate value of the property, the synergistic program that can still continue executing multiple times is called the generator.

Note: While iterators do not have the above characteristics and are not suitable for dealing with some huge class sequence objects, it is recommended that you prioritize scenarios that use generators to handle iterations.

Benefits of the generator

In summary: The best scenario for using generators is when you need to iterate through a huge collection of data. For example: a huge file/a complex database query and so on.

EXAMPLE 2: Read a large file

def read_file(fpath):     1024     with‘rb‘as f:         whileTrue:             block = f.read(BLOCK_SIZE)             if block:                 yield block             else:                 return

Calling the Read () method directly on a file object causes unpredictable memory consumption. A good approach is to use fixed-length buffers to continuously read portions of the file. With yield, we no longer need to write an iterative class of read files, so we can easily implement file reads.

Enhanced generator Features

In addition to using next() methods to get the next generated value, the user can also use send() a method to return a new or modified value to the generator. In addition, you can use the close() method to exit the generator at any time.
EXAMPLE 3:

in [5]: def counter(start_at=0):...: Count = start_at ...: while True:...: val = (yieldCount) ...:ifVal is  not None: ....: Count = val ...:Else:...: Count + =1...: in [6]: Count = Counter (5) in [7]: Type (count) out[7]: Generatorin [8]: Count.next () out[8]:5in [9]: Count.next () out[9]:6in [Ten]: Count.send (9)# Returns a new value to yield count in the generatorout[Ten]:9in [ One]: Count.next () out[ One]:Tenin [ A]: Count.close ()# Close a generatorin [ -]: Count.next ()---------------------------------------------------------------------------stopiteration Traceback (most recent) <ipython-input- --3963Aa0a181a>inch<module> ()---->1Count.next () Stopiteration:

Builder expression

The builder expression is an extension of list resolution, as described above: The generator is a specific function that allows you to return an intermediate value, then suspends execution of the code and resumes execution later. The disadvantage of list parsing is that it must generate all the data at once to create the list object, so it is not suitable for iterating through large amounts of data.
The generator expression solves this problem by combining list parsing and generator.

• List parsing
  [expr for iter_var in iterable if cond_expr]

• Builder expression
  (expr for iter_var in iterable if cond_expr)

The syntax of the two is very similar, but the generator expression returns not a list type object, but a generator object, and the generator is a memory-friendly structure.

Example of generator expression model

See the benefits of the generator by improving the functionality of finding the longest line in a file .
EXAMPLE 4 : A more common method of assigning a longer line to a variable longest by looping.

f = open(‘FILENAME‘‘r‘0whileTrue:    linelen = len(f.readline().strip())    ifnot linelen:        break    if linelen > longest:        longest = linelenf.close()return longest

Obviously, in this example, the object that needs to be iterated is a file object.

Improvement 1:
It is important to note that if we read all the rows of a file, we should release the file resource as soon as possible. For example: A log file, there will be a lot of different processes to operate it, so we can not tolerate any one process holding the file handle not put.

f = open(‘FILENAME‘‘r‘0allLines = f.readlines()f.close()forin allLines:    linelen = len(line.strip())    ifnot linelen:        break    if linelen > longest:        longest = linelenreturn longest

Improvement 2:
We can use list parsing to simplify the above code, for example: Each row is processed when it gets a list of all the alllines rows.

f = open(‘FILENAME‘‘r‘0forin f.readlines()]f.close()forin allLines:    linelen = len(line)    ifnot linelen:        break    if linelen > longest:        longest = linelenreturn longest

Improvement 3:
When we're dealing with a huge file, it's file.readlines() not a sensible choice because it readlines() reads all the lines in the file. So do we have any other way to get a list of all the rows? We can apply the built-in iterators of the file files.

f = open(‘FILENAME‘‘r‘forin f]f.close()return max(allLinesLen)   # 返回列表中最大的数值

It is no longer necessary to use the method of circular comparison and retain the current maximum value to process, the last element of all the length of the row is stored in the list object, and then get the larger value.

Improvement 4:
The problem here is that when using list parsing to process a file object, all of the file's rows are read into memory, and then a new list object is created, which is an unfriendly implementation of memory. Then we can use a generator expression instead of list parsing.

f = open(‘FILENAME‘‘r‘forin f)   # 这里的 x 相当于 yield xf.close()return max(allLinesLen)

Because if you use a generator expression as a parameter in a function , we can omit the parentheses ' () ' so that the code will be further simplified:

f = open(‘FILENAME‘‘r‘forin f)f.close()return longest

finally : We are able to implement this function in one line of code, allowing the Python parser to process the open file.
Of course not that the less code, the better, for example, the following line of code each cycle will call a open() function, the efficiency is not improved 4 higher.

returnforin open(‘FILENAME‘))

Summary

When iterating through an object, we should prioritize using a generator substitution iterator and using a generator expression instead of a list resolution. Of course, this is not absolute. iterators and generators are important features of Python and have a good understanding of how to write more pythonic code.

Python Advanced _ Builder & Builder expression