Python iterator and Python Generator

Python iterator and Python Generator

The problem is that it is generated during the loop in Python. anyone familiar with Python knows that it is not similar to the for loop in other languages and can only perform loop traversal through the for in method. The most typical application is to use the range Function to generate a list and then use the for in operation, as shown below:

Copy codeThe Code is as follows:
#! /Usr/bin/env python
For I in range (10 ):
Print I

The meaning of the code is very easy to understand. range will generate a list and traverse it using the for in most list. It is similar to for (I = 0; I <n; I ++) for the same effect, the details of the range function can be viewed here. The problem is that the range object will generate a list, so the content of this list must be stored in the memory. When the number of loops required is too large, it will occupy a considerable amount of memory, in order to count the memory usage of range, I used range for 6 times to generate a list of, and lengths, and then counted the memory usage:
Copy codeThe Code is as follows:
Memory occupied by test code
Range (100) 2.0 MB
Range (10000) 2.2 MB
Range (100000) 3.8 MB
Range (1000000) 19.5 MB
Range (10000000) 168.5 MB
Range (100000000) 1465.8 MB

As you can see, as the base number increases, the memory usage increases exponentially. Obviously, when performing large loop operations, avoid using range.

To solve the above problem, python provides another function xrange, which is very similar to range, but occupies much smaller memory than range. For more information, see here, the memory usage of objects generated by xrange is almost unchanged no matter how many parameters are used. The question is, how is xrange implemented internally? Why is it so different from range performance? To verify my conjecture, first try to use python to implement the function zrange similar to xrange:
Copy codeThe Code is as follows:
#! /Usr/bin/env python
Class zrange (object ):
Def _ init _ (self, stop ):
Self. _ pointer = 0
Self. stop = stop
Def _ iter _ (self ):
Return self
Def next (self): # In python3.0, use _ next _ instead __
If self. _ pointer> = self. stop:
Raise StopIteration
Else:
Self. _ pointer = self. _ pointer + 1
Return self. _ pointer-1
Test = zrange (10000000)
For I in test:
Print I

The running result is the same as that of xrange. The zrange memory usage test shows that, like xrange, the parameter size has almost no effect on memory usage. So what is the difference between it and range?

As mentioned above, range produces a list, and both the custom zrange and the system's built-in xrange generate an object, such as an object generated by xrange or zrange, it is called an iteratable object. It provides an external method to traverse its internal elements without worrying about its internal implementation. In the above zrange implementation, the most critical implementation is to establish an internal pointer _ pointer, which records the current access location, the next access can be performed through the pointer status.

In Python or other languages, there are many similar methods to accessing object content through iteration, such as reading content in a file:
Copy codeThe Code is as follows:
#! /Usr/bin/env python
F = open ('zrange. py', 'R ')
While True:
Line = f. readline ()
If not line:
Break
Print line. strip ()
F. close ()

We all know that using readline saves resources than reandlines. In fact, readline and readlines are similar to xrange and range. One is to record the current position through the pointer and move the Pointer Forward to a unit during the next access, the other is to directly store all the content in the memory. In file operation functions, you can also manually adjust the pointer position through seek to skip or read some content repeatedly.

It can be said that in the implementation of the iterator, internal pointers are the key to saving resources and making the iteration run normally.