Using Python for concurrent programming

Making computer programs run concurrently is a topic that is often discussed, and today I want to talk about the various concurrency patterns under Python.

concurrency mode

Threads (thread)

Multithreading is almost every program ape in the use of each language will first think of the tool used to solve concurrency (JS programmer please avoid), using multithreading can effectively utilize CPU resources (Python exception). However, the complexity of the program brought by multithreading is unavoidable, especially the synchronization problem of competing resources.

However, in Python because of the use of the global interpretation of the Lock (GIL), the code can not concurrently run on multiple cores concurrently, that is, Python multithreading can not be concurrent, many people will find the use of multithreading to improve their own Python code, the operation of the program has decreased efficiency, This is a matter of how the egg hurts! If you want to know more details, we recommend reading this article. It is difficult to actually use a multithreaded programming model, which is not a programmer's fault, because parallel thinking is anti-human, most of us are serial (schizophrenia is not discussed), and the computer architecture of von Neumann design is based on sequential execution. So if you can't always get your multithreaded program done, congratulations, you're a normal-thinking program ape:)

Python provides interfaces for two sets of threads, one for the thread module, and a basic, low-level interface, using function as the running body of the thread. Another group is the threading module, which provides an easier-to-use object-based interface (similar to Java), inherits thread objects to implement threads, and provides other thread-related objects, such as Timer,lock

Examples of using the thread module

1

2

3

4

5

Import Thread

def worker ():

"" Thread worker Function "" "

PRint ' Worker '

Thread.start_new_thread (worker)

Examples of using threading modules

1

2

3

4

5

6

Import threading

def worker ():

"" Thread worker Function "" "

print ' Worker '

t = Threading. Thread (Target=worker)

T.start ()

or Java Style

1

2

3

4

5

6

7

8

9

10

Import threading

Class Worker (threading. Thread):

def __init__ (self):

Pass

def run ():

"" Thread worker Function "" "

print ' Worker '

T = Worker ()

T.start ()

Processes (Process)

Due to the problem of global interpretation of locks mentioned earlier, Python's better parallel approach is to use multi-process, which makes it very efficient to use CPU resources and achieve real concurrency. Of course, the cost of the process is larger than the thread, meaning that if you want to create a staggering number of concurrent processes, you need to consider whether your machine has a strong heart.

Python's mutliprocess module and threading have similar interfaces.

1

2

3

4

5

6

7

8

From multiprocessing import Process

def worker ():

"" Thread worker Function "" "

print ' Worker '

p = Process (Target=worker)

P.start ()

P.join ()

Because threads share the same address space and memory, communication between threads is very easy, but communication between processes is a bit more complicated. Common interprocess communication has, pipelines, Message Queuing, socket interfaces (TCP/IP), and so on.

The Python mutliprocess module provides a packaged pipeline and queue for easy delivery of messages between processes.

Synchronization between Python processes uses locks, which are the same for drinking threads.

In addition, Python provides a process pool object that allows you to easily manage and control threads.

Remote distributed Host (distributed Node)

With the advent of the Big Data era, Moore's theorem seems to have lost its effect on a single machine, the calculation and processing of data need distributed computer network to run, program parallel running on multiple host nodes, is now the software architecture must consider the problem.

There are several common ways of interprocess communication between remote hosts

TCP/IP

TCP/IP is the basis of all remote communication, but the API is relatively low level, it is cumbersome to use, so generally do not consider

Remote method calls to the remotes Function call

RPC is a means of early communication between remote processes. There is an open-source implementation under Python RPYC

Remote Objects Remotely Object

The remote object is a higher-level package, and the program can manipulate a local agent for a remote object as if it were a native object. The most widely used specification for remote objects Corba,corba The greatest benefit is the ability to communicate in different languages and platforms. When there are some other implementations of remote objects, such as the Rmi,ms of Java, in the languages and platforms that are not used

Python's open source implementation, with many support for remote objects

Dopy

Fnorb (CORBA)

ICE

Omniorb (CORBA)

Pyro

YAMI

Message Queuing Messages Queue

Messages are a more flexible means of communication than RPC or remote objects, and a common message mechanism that supports the Python interface is

RabbitMQ

ZeroMQ

Kafka

AWS SQS + BOTO

There is no significant difference between executing concurrent and on-premises multiple processes on a remote host, and there is a need to resolve inter-process communication issues. Of course, the management and coordination of remote processes is more complex than local.

Python has many open-source frameworks to support distributed concurrency and provides effective management tools including:

Celery

Celery is a very mature Python distributed framework that can perform tasks asynchronously in a distributed system and provide effective management and scheduling capabilities. Refer here

SCOOP

SCOOP (Scalable COncurrent Operations in Python) provides an easy-to-use, distributed calling interface for concurrency using the future interface.

Dispy

Provides more lightweight distributed parallel services compared to celery and scoop,dispy

Pp

PP (Parallel python) is another lightweight Python parallel service, refer to Here

Asyncoro

Asyncoro is another Python framework that uses generator to implement distributed concurrency.

Of course there are many other systems that I did not list

In addition, many distributed systems provide support for Python interfaces, such as Spark

Pseudo-Threading (Pseudo-thread)

There is also a concurrency method is not common, we can call it pseudo-thread, that is, it looks like a thread, the interface is similar to the thread interface, but the actual use of non-threading, the corresponding thread cost does not exist.

Greenlet

Greenlet provides lightweight coroutines to support intra-process concurrency.

Greenlet is a byproduct of stackless, using Tasklet to support a technique called micro-threading (Mirco-thread), an example of a pseudo-thread using Greenlet

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

From Greenlet import Greenlet

def test1 ():

Print 12

Gr2.switch ()

Print 34

Def test2 ():

Print 56

Gr1.switch ()

Print 78

Gr1 = Greenlet (test1)

GR2 = Greenlet (test2)

Gr1.switch ()

Run the above program to get the following results:

1

2

3

12

56

34

The pseudo-thread GR1 switch prints 12 and then calls GR2 switch to get 56, then switch back to Gr1, print 34, and then the pseudo-thread gr1 ends, and the program exits, so 78 will never be printed. From this example we can see that using pseudo-threading, we can effectively control the execution process of the program, but the pseudo-thread does not exist in the true sense of concurrency.

Both Eventlet,gevent and Concurence are based on Greenlet to provide concurrency.

Eventlet http://eventlet.net/

Eventlet is a Python library that provides network calls and can be used in a non-blocking manner to invoke blocking IO operations.

1

2

3

4

5

6

7

8

9

10

11

12

Import Eventlet

From Eventlet.green import urllib2

URLs = [' http://www.google.com ', ' http://www.example.com ', ' http://www.python.org ']

def fetch (URL):

return Urllib2.urlopen (URL). Read ()

Pool = Eventlet. Greenpool ()

For body in Pool.imap (Fetch, URLs):

Print ("Got Body", Len (body))

Execution results are as follows

1

2

3

(' Got body ', 17629)

(' Got body ', 1270)

(' Got body ', 46949)

Eventlet the URLLIB2 has been modified to support the generator operation, the interface and URLLIB2 are consistent. The greenpool here is consistent with the Python pool interface.

Gevent

Gevent and Eventlet similar, about their differences you can refer to this article

1

2

3

4

5

6

7

Import Gevent

From gevent import socket

URLs = [' www.google.com ', ' www.example.com ', ' www.python.org ']

Jobs = [Gevent.spawn (socket.gethostbyname, url) for URL in URLs]

Gevent.joinall (Jobs, timeout=2)

Print [Job.value for job in jobs]

The results of the implementation are as follows:

1

[' 206.169.145.226 ', ' 93.184.216.34 ', ' 23.235.39.223 ']

Concurence https://github.com/concurrence/concurrence

Concurence is another open source library that uses Greenlet to provide network concurrency, which I have not used, and you can try it out for yourself.

Practical use

Usually need to use the concurrency of two, one is computationally intensive, that is, your program requires a lot of CPU resources, the other is IO-intensive, the program may have a large number of read and write operations, including read and write files, send and receive network requests and so on.

COMPUTE-intensive

For compute-intensive applications, we use the famous Monte Carlo algorithm to calculate PI values. The rationale is as follows

The Monte Carlo algorithm uses statistical principles to simulate the calculation of pi, in which the probability of a random point falling in a 1/4-circle area (red dot) is proportional to its area. The probability p = Pi * R*R/4: r* R, where R is the side length of the square, the radius of the circle. That is, the probability is pi 1/4, using this conclusion, as long as we can simulate the probability of the point falling on the One-fourth circle to know the Pi, in order to get this probability, we can through a large number of experiments, that is to generate a large number of points, to see where this point in the region, and then the results of statistics.

The basic algorithm is as follows:

1

2

3

4

5

From math import Hypot

From random import random

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Here the test method does the N (tries) test, returning the number of points falling in the One-fourth circle. The method of judging is to check the distance from the point to the center of the circle, if it is less than R.

With a lot of concurrency, we can run multiple tests quickly, the more times we experiment, the closer we get to real pi.

The program code for different concurrency methods is given here

Non-concurrency

We start with a single thread, but the process runs to see how performance

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

From math import Hypot

From random import random

Import Eventlet

Import time

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

result = Map (test, [tries] * Nbfutures)

RET = 4. * SUM (Result)/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

Print CALCPI (3000,4000)

Multithreading thread

In order to use the thread pool, we use multiprocessing's dummy package, which is an encapsulation of multithreading. Note that the code here does not refer to a thread in a word, but it is true to multithreading.

By testing our discovery of the (Jing) Heart (ya), as expected, when the thread pool is 1, it runs as well as no concurrency, and when we set the thread pool number to 5 o'clock, it takes almost no concurrency twice times, my test data from 5 seconds to 9 seconds. So for computationally intensive tasks, let's give up on multithreading.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21st

From Multiprocessing.dummy import Pool

From math import Hypot

From random import random

Import time

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

p = Pool (1)

result = P.map (test, [tries] * Nbfutures)

RET = 4. * SUM (Result)/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

if __name__ = = ' __main__ ':

p = Pool ()

Print ("PI = {}". Format (CALCPI (3000, 4000))

Multi-Process Multiprocess

In theory for computationally intensive tasks, using multi-process concurrency is appropriate, in the following example, the size of the process pool is set to 5, modify the size of the process pool can see the impact on the results, when the process pool is set to 1 o'clock, and the results of multithreading is similar to the time, because there is no concurrency; When set to 2 o'clock, There was a noticeable improvement in response time, which was no more than half the previous concurrency, while the continued expansion of the process pool had little or no impact on performance, and perhaps my Apple air CPU had only two cores.

Beware, if you set up a very large process pool, you will encounter Resource temporarily unavailable error, the system does not support the creation of too many processes, after all, resources are limited.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

From multiprocessing import Pool

From math import Hypot

From random import random

Import time

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

p = Pool (5)

result = P.map (test, [tries] * Nbfutures)

RET = 4. * SUM (Result)/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

if __name__ = = ' __main__ ':

Print ("PI = {}". Format (CALCPI (3000, 4000))

Gevent (pseudo-threading)

Whether it is gevent or eventlet, because there is no actual concurrency, the response time and no concurrency difference is small, this and test results are consistent.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

Import Gevent

From math import Hypot

From random import random

Import time

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

Jobs = [Gevent.spawn (test, T) for T in [tries] * Nbfutures]

Gevent.joinall (Jobs, timeout=2)

RET = 4. * SUM ([Job.value for job in jobs])/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

Print CALCPI (3000,4000)

Eventlet (pseudo-threading)

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

From math import Hypot

From random import random

Import Eventlet

Import time

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

Pool = Eventlet. Greenpool ()

result = Pool.imap (test, [tries] * Nbfutures)

RET = 4. * SUM (Result)/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

Print CALCPI (3000,4000)

SCOOP

The future interface in scoop conforms to the definition of PEP-3148, which is the future interface provided in Python3.

In the default Scoop configuration environment (single machine, 4 worker), concurrency performance is improved, but not as many processes as two process pool configurations.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

From math import Hypot

From random import random

From Scoop Import Futures

Import time

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

Expr = Futures.map (test, [tries] * Nbfutures)

RET = 4. * SUM (expr)/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

if __name__ = = "__main__":

Print ("PI = {}". Format (CALCPI (3000, 4000))

Celery

Task code

1

2

3

4

5

6

7

8

9

10

11

From celery import celery

From math import Hypot

From random import random

App = celery (' tasks ', backend= ' amqp ', broker= ' amqp://guest@localhost//')

App.conf.CELERY_RESULT_BACKEND = ' Db+sqlite:///results.sqlite '

@app. Task

def test (tries):

return sum (Hypot (random (), random ()) < 1 for _ in range (tries))

Client code

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

From celery Import Group

From Tasks Import test

Import time

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

result = Group (TEST.S (tries) for I in Xrange (Nbfutures)) (). Get ()

RET = 4. * SUM (Result)/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

Print CALCPI (3000, 4000)

The results of the concurrency test using celery are unexpected (the environment is single-machine, 4frefork concurrency, message broker is RABBITMQ), the worst of all test cases, and the response time is no concurrency 5~6 times. This may be because the overhead of controlling coordination is too high. For such a computational task, celery may not be a good choice.

Asyncoro

Asyncoro test results are consistent with non-concurrency.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

Import Asyncoro

From math import Hypot

From random import random

Import time

def test (tries):

Yield sum (hypot (random (), random ()) < 1 for _ in range (tries))

Def CALCPI (Nbfutures, tries):

ts = Time.time ()

Coros = [Asyncoro. Coro (test,t) for T in [tries] * Nbfutures]

RET = 4. * SUM ([Job.value () for job in Coros])/float (nbfutures * tries)

span = Time.time ()-TS

Print "Time spend", span

return ret

Print CALCPI (3000,4000)

IO-intensive

IO-intensive tasks are another common use case, such as a network Web server, which is an example of how many requests can be processed per second for a Web server's important metrics.

We'll take the web as the simplest example.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

From math import Hypot

Import time

Import Urllib2

URLs = [' http://www.google.com ', ' http://www.example.com ', ' http://www.python.org ']

def test (URL):

return Urllib2.urlopen (URL). Read ()

def testio (nbfutures):

ts = Time.time ()

Map (test, URLs * nbfutures)

span = Time.time ()-TS

Print "Time spend", span

Testio (10)

The code under the different concurrency libraries, I don't list them, because they're similar. You can refer to the code in the calculation-intensive.

Through testing we can find that for IO-intensive tasks, using multi-threading, or multi-process can effectively improve the efficiency of the program, and the use of pseudo-threading performance is very significant, eventlet than without concurrency, the response time from 9 seconds to 0.03 seconds. At the same time, Eventlet/gevent provides a non-blocking asynchronous invocation mode, which is very convenient. It is recommended to use threads or pseudo-threads, because threads and pseudo-threads consume less resources in similar response times.

Summarize

Python provides different concurrency methods that correspond to different scenarios, and we need to choose different ways to do concurrency. Choose the right way, not only to understand the principle of the method, but also to do some testing and testing, data is the best reference for you to choose.

The above is the use of Python for concurrent programming content, more related articles please pay attention to topic.alibabacloud.com (www.php.cn)!

