Describes the usage of the yield generator in Python.

Describes the usage of the yield generator in Python.
Yield is the meaning of generation, but in python it is understood as a generator. The use of the generator can mainly be iterated, which simplifies many calculation models (not very familiar with how to simplify ).

Yield is an expression with a returned value.

When a function contains yield, it is no longer a common function, but a generator. When the function is called, it is not automatically executed, but paused,

Reference: http://www.aichengxu.com/view/64610

See the First example:

Example 1:

>>> Def mygenerator ():... print 'start... '... yield 5... >>> mygenerator () // called here, and no start... is printed... it indicates that the yield function is not run, that is, the <generator object mygenerator at 0xb762502c >>> mygenerator (). next () // call next () to run the function. start... 5 >>>

If multiple yield instances exist in a function, next () stops before the next yield instance. See Example 2:

Example 2:

>>> Def mygenerator ():... print 'start... '... yield 5... >>> mygenerator () // called here, and no start... is printed... it indicates that the yield function is not run, that is, the <generator object mygenerator at 0xb762502c >>> mygenerator (). next () // call next () to run the function. start... 5 >>>

Why does yield 5 output 5 and yield 23 output 23?

We assume that yield is an expression and the returned value exists.

Can I assume that the returned value of yield 5 must be 5? In fact, this is not the case. This function has a certain relationship with the send function, which is essentially similar to next (). The difference is that send is used to pass the value of the yield expression, however, next cannot pass a specific value, but can only pass None. Therefore, g. next () and g. send (None) is the same. See example 3:

Example 3:

>>> Def fun ():... print 'start... '... m = yield 5... print m... print 'midddle... '... d = yield 12... print d... print 'end... '... >>> m = fun () // create an object >>> m. next () // execute the function to start before the next yield... 5 >>> m. send ('message') // use send () to pass the value message // send () to pass in the middle... 12 >>> m. next () None // visible next () The returned value is null end... traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration

Use in multiprocess

When processing data in python, memory-heavy data often results in the absence of Reverse Running of the program or the efficiency of other programs on the server during running. In this case, the data set is usually traversed through genertor.

But as we know, generoter seems to be only consumed by a single process, which is very inefficient.

Generator can be consumed by pool. map.

Let's take a look at the source code of pool. py.

for i, task in enumerate(taskseq):  ...  try:   put(task)  except IOError:   debug('could not put task on queue')   break

Actually, all generator consumption is first put into the queue. Then run the map command in parallel. This solves the problem of using map for parallelism.

But it still does not solve the problem of memory usage. Here, two steps occupy the memory.

The first step is to completely consume the generator.

Step 2: Perform parallel operations on all data.

Solve the first problem by consuming generator.

The second problem can be solved through imap.

The sample code is as follows:

import multiprocessing as mpimport itertoolsimport timedef g(): for el in xrange(50):  print el  yield elimport osdef f(x): time.sleep(1) print str(os.getpid()) +" "+ str(x) return x * xif __name__ == '__main__': pool = mp.Pool(processes=4)    # start 4 worker processes go = g() result = [] N = 11 while True:  g2 = pool.imap(f, itertools.islice(go, N))  if g2:   for i in g2:    result.append(i)    time.sleep(1)  else:   break print(result)

Ps: usage considerations. When producing CE data, do as few operations as possible, so that even map is a single thread to consume data. So try to put the operation in map. In this way, we can make better use of multiple processes to improve efficiency.

Python learning tutorial center: http://www.aichengxu.com/item/15

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.