Python Data Reading: Generator

Python generator are a simple way to create iterators.

In simple terms, a generator is a function that returns an object (iterator) that we can iterate, and an iterator returns one value at a time

Compared to using lists to load all data into memory, the generator saves a lot of memory space.

The data reading part of deep learning generally requires the use of iterators.

Create a generator

There are two ways to create a generator:

List generation formula [] changed to ()

Use the yield method

First. List generation formula [] changed to ()

Change the [] to () in the list generator to create a generator

>>> L = [x * x for x in range(10)]

>>> L[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

>>> g = (x * x for x in range(10))

>>> g<generator object <genexpr> at 0x1022ef630>


The difference between creating L and g is only the outermost [] and (), L is a list, and g is a generator.

Second. Using the yield method

The generator https://edu.alibabacloud.com/certification/clouder_sparkpython?spm=a2c61.500110.200003.2.4b247f1b2NtqkEexplained:

1, yield: used to return data. After the program executes to yield, return the result, remember the current state, suspend execution, and the next time, according to the previous state, return to the next result, remember the new state, suspend execution ,...

To put it simply, it is called once, spit out 1 data, then called again, spit out the second data, and called again, spit out the third..., when the model is trained, each batch of data is spit through the yield from;

2. The generator can spit out data one by one by calling the next function, or it can call its own __next__ function;

3. The generator is a special iterator that can traverse the data through the for loop;

# Define the generator function def my_gen(): --my_list = ['I','love','python', 123]--for item in my_list:----yield item# Get the generator gen1 = my_gen( )# Get the data in the generator print(gen1.__next__()) # Iprint(next(gen1)) # love# Define the generator and iterate over its data gen2 = my_gen() for item in gen2:--print(item )


Third. Function definition generator

Define a function externally, define a generator function internally, and the external function returns the name of the generator function

def outer_fn(num=10):--def inner_gen(para=num):----for item in range(para + 1):------yield item ** 2----return inner_geninner_fn = outer_fn(5) # external function execution, return internal function gen = inner_fn() # internal function execution, return generator print(next(gen)) # 0print(next(gen))#1