Concurrent programming with generators implemented in Python

Source: Internet
Author: User
Tags generator thread

This article introduces the simple introduction of Python in the use of the generator implementation of concurrent programming, the use of yield generator function for multiple process programming is a Python learning the important knowledge of the advanced, the need for friends can refer to the

We all know that concurrent (not parallel) programming currently has four ways, multiple processes, multiple threads, Asynchrony, and coprocessor.

Multi-process programming has a C-like os.fork in Python, and, of course, a higher-layer multiprocessing standard library, providing a similar nginx master process and worker in a previously written Python highly available programming approach The method of signal processing between processes ensures that the exit of the business process can be perceived by the main process.

Multithreaded programming python has thread and threading, and the so-called threads in Linux are actually LWP lightweight processes that have the same scheduling as processes in the kernel, about Lwp,cow (write-time copies), Fork,vfork, Clone and so on more information, here no longer repeat.

There are three implementations of asynchronous Select,poll,epoll in Linux, and asynchronous is not the focus of this article.

To say that the yield is definitely going to say, let's take a look at an example:


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 #coding =utf-8 Import time Import sys # producer def produce (L): i=0 while 1:if i < 5:l.append (i) yield I i=i+1 time.sleep ( 1 Else:return # Consumer def consume (l): p = Produce (L) while () while Len (l) > 0:print l.pop () except St OpIteration:sys.exit (0) L = [] consume (L)

In the above example, when the program executes to the yield I of produce, a generator is returned, and when we call () in custom, the program returns to produce I continue to execute, so l yield the element. Then we print l.pop () until () throws a Stopiteration exception.

We can see from the example above that the scheduling of the coprocessor is not visible to the kernel, and the coordination between the processes is coordinated, which makes the concurrent volume in the tens of thousands of times, the performance of the coprocessor is far higher than the thread.


1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Import stackless Import urllib2 def output (): While 1:url=chan.receive () Print URL f=urllib2.urlopen (URL) #print () Print stackless.getcurrent () def input (): F=open (' Url.txt ') l=f.readlines () for I in L:chan.send (i) Chan=stackless.cha Nnel () [Stackless.tasklet (output) () for I in Xrange ()] Stackless.tasklet (input) () ()

About the process, you can refer to the implementation of Greenlet,stackless,gevent,eventlet.

Related Article

E-Commerce Solutions

Leverage the same tools powering the Alibaba Ecosystem

Learn more >

Apsara Conference 2019

The Rise of Data Intelligence, September 25th - 27th, Hangzhou, China

Learn more >

Alibaba Cloud Free Trial

Learn and experience the power of Alibaba Cloud with a free trial worth $300-1200 USD

Learn more >

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.