Describes the usage of the yield generator in Python.

Source: Internet
Author: User
Tags imap

Describes the usage of the yield generator in Python.
Yield is the meaning of generation, but in python it is understood as a generator. The use of the generator can mainly be iterated, which simplifies many calculation models (not very familiar with how to simplify ).

Yield is an expression with a returned value.

When a function contains yield, it is no longer a common function, but a generator. When the function is called, it is not automatically executed, but paused,

Reference: http://www.aichengxu.com/view/64610

See the First example:

Example 1:

>>> Def mygenerator ():... print 'start... '... yield 5... >>> mygenerator () // called here, and no start... is printed... it indicates that the yield function is not run, that is, the <generator object mygenerator at 0xb762502c >>> mygenerator (). next () // call next () to run the function. start... 5 >>>


If multiple yield instances exist in a function, next () stops before the next yield instance. See Example 2:

Example 2:

>>> Def mygenerator ():... print 'start... '... yield 5... >>> mygenerator () // called here, and no start... is printed... it indicates that the yield function is not run, that is, the <generator object mygenerator at 0xb762502c >>> mygenerator (). next () // call next () to run the function. start... 5 >>>


Why does yield 5 output 5 and yield 23 output 23?

We assume that yield is an expression and the returned value exists.

Can I assume that the returned value of yield 5 must be 5? In fact, this is not the case. This function has a certain relationship with the send function, which is essentially similar to next (). The difference is that send is used to pass the value of the yield expression, however, next cannot pass a specific value, but can only pass None. Therefore, g. next () and g. send (None) is the same. See example 3:

Example 3:

>>> Def fun ():... print 'start... '... m = yield 5... print m... print 'midddle... '... d = yield 12... print d... print 'end... '... >>> m = fun () // create an object >>> m. next () // execute the function to start before the next yield... 5 >>> m. send ('message') // use send () to pass the value message // send () to pass in the middle... 12 >>> m. next () None // visible next () The returned value is null end... traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration


Use in multiprocess

When processing data in python, memory-heavy data often results in the absence of Reverse Running of the program or the efficiency of other programs on the server during running. In this case, the data set is usually traversed through genertor.

But as we know, generoter seems to be only consumed by a single process, which is very inefficient.

Generator can be consumed by pool. map.

Let's take a look at the source code of pool. py.

for i, task in enumerate(taskseq):  ...  try:   put(task)  except IOError:   debug('could not put task on queue')   break


Actually, all generator consumption is first put into the queue. Then run the map command in parallel. This solves the problem of using map for parallelism.

But it still does not solve the problem of memory usage. Here, two steps occupy the memory.

The first step is to completely consume the generator.

Step 2: Perform parallel operations on all data.

Solve the first problem by consuming generator.

The second problem can be solved through imap.

The sample code is as follows:

import multiprocessing as mpimport itertoolsimport timedef g(): for el in xrange(50):  print el  yield elimport osdef f(x): time.sleep(1) print str(os.getpid()) +" "+ str(x) return x * xif __name__ == '__main__': pool = mp.Pool(processes=4)    # start 4 worker processes go = g() result = [] N = 11 while True:  g2 = pool.imap(f, itertools.islice(go, N))  if g2:   for i in g2:    result.append(i)    time.sleep(1)  else:   break print(result)


Ps: usage considerations. When producing CE data, do as few operations as possible, so that even map is a single thread to consume data. So try to put the operation in map. In this way, we can make better use of multiple processes to improve efficiency.

Python learning tutorial center: http://www.aichengxu.com/item/15

Copyright Disclaimer: This article is an original article by the blogger and cannot be reproduced without the permission of the blogger.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.