Parallel Python concurrent development of multi-core parallel programs

Python is an interpreted language, while the python interpreter uses Gil (Global interpreter lock) to prohibit parallel execution internally, this Gil limits that you can only execute one bytecode command at a time on a multi-core processor. python 3.0 has been improved. By default, a multi-processor programming library is available. python2.xx is not supported yet.

The parallel Python library is designed for secondary use. It can not only work collaboratively with multi-core processors, but also run through network clusters.

Http://www.parallelpython.com/

The following is a Chinese introduction from here:

1. Introduction

PP is a python module that provides a mechanism for parallel execution of Python code on SMP (multiple CPUs or multiple cores) and clusters (multiple computers connected through the network. Lightweight, easy to install, and integrated with other software. PP is also a cross-platform, open source module implemented with pure Python code.

2. Functions

* Execute Python code in parallel on SMP and Cluster
* A Work-based parallel mechanism that is easy to understand and implement to convert a walk-through application into a parallel
* Automatically construct the optimal configuration (the number of worker processes equals the number of system processors by default)
* Dynamic Processor Allocation (allows changing the number of worker processors at runtime)
* Function working cache (transparent cache mechanism ensures that subsequent calls reduce load)
* Dynamic Load Balancing (tasks are dynamically allocated to various processors)
* Sha-based connection encryption authentication
* Cross-platform porting (Windows/Linux/Unix)
* Open Source Code

3. Open the engine

Modern Python programs have been widely used in business logic, data analysis and scientific computing. SMP (multi-processor or multi-core) and cluster (multiple computing machines connected through the network) are widely used, and the Python code that needs to be executed in parallel in the market.
The simplest way to write parallel programs on SMP computers is to use multithreading. Even so, the use of the 'thread' and 'threading' modules still cannot implement parallelism at the bytecode level. Because the python interpreter uses Gil (Global interpreter lock) to prohibit parallel execution internally. This Gil limits that you can only execute one bytecode command at a time on the SMP machine.
To solve this problem, the PP module provides a simple way to implement parallel Python applications. Ppsmp internally uses processes and IPC (inter-process communication) to organize parallel computing. After all the internal details and complexity are processed, your application only needs to submit work tasks and retrieve the results. This is also the simplest method for writing parallel programs.
For better implementation, all software using PP can connect and collaborate over the network. Cross-platform and Dynamic Load Balancing allow PP to easily organize multi-platform and heterogeneous cluster computing environments.

Take the official example:

#-*-Coding: UTF-8-*-# features # Name: # purpose: # Author: ankier # created: 05-02-2013 # copyright: (c) ankier 2013 # licence: <your licence> # calculate import math, sys, timeimport ppdef isprime (n): "returns whether N is a prime number" if not isinstance (n, INT ): raise typeerror ("argument passed to is_prime is not of 'int' type") if n <2: Return false if n = 2: Return true max = int (math. ceil (math. SQRT (N) I = 2 while I <= max: If n % I = 0: Return false I + = 1 return truedef sumprimes (n ): "Calculate the sum of all prime numbers from 2 to N" Return sum ([X for X in xrange (2, n) If isprime (x)]) inputs = (100000,100, 100200,100, 100400,100, 100600,100,) start_time = time. time () for input in inputs: Print sumprimes (input) print 'single-threaded execution, total consumed time', time. time ()-start_time,'s '# tuple of all parallel Python servers to connect withppservers = () # ppservers = ("10.0.0.1",) If Len (sys. argv)> 1: ncpus = int (sys. argv [1]) # creates jobserver with ncpus workers job_server = pp. server (ncpus, ppservers = ppservers) else: # creates jobserver with automatically detected number of workers job_server = pp. server (ppservers = ppservers) print "Number of worker cores available for PP", job_server.get_ncpus (), "Workers" start_time = time. time () jobs = [(input, job_server.submit (sumprimes, (input,), (isprime,), ("math",) for input in inputs] for input, job in jobs: Print "sum of Primes below", input, "is", JOB () print "execution time in multiple threads:", time. time ()-start_time, "S" job_server.print_stats ()

Running result:

454396537454996777455898156456700218457603451458407033459412387460217613 single-threaded execution, total time consumption 3.15899991989 spp number of core working threads available 4 workerssum of Primes below 100000 is 454396537sum of Primes below 100100 is 454996777sum of Primes below 100200 is of Primes below 100300 is 456700218sum of Primes below 100400 is 457603451sum of Primes below 100500 is 458407033sum of Primes below 100600 is 459412387sum of Primes below 100700 is 460217613 execution time in multiple threads: 1.44400000572 sjob execution statistics: job count | % of all jobs | job time sum | time per job | job Server 8 | 100.00 | 5.2460 | 0.655750 | localtime elapsed since server creation 1.444000005720 active tasks, 4 Cores