Try cython and openmp

1. Original Intention

Recently, I learned to use python. python is a programming language for programmers. It is very convenient to write and greatly saves development efficiency. Moreover, small-scale programs have good running efficiency. I wrote a blog post "[summary] Using python to write programs" two days ago, which greatly praised python. But in the past two days, I was "cursed. The data scale is a little large, and the execution efficiency of python is poor. A program written in the past two days, even though I have made Optimizations in the python language within what I know, has been running for five hours. I want to change the program to c ++, but it takes a long time to develop and may be changed in the future. So for now.

I checked the python efficiency on the Internet. On the one hand, there are not many materials on the Internet. For example, we all know that the set in stl is implemented by the red and black trees. However, it seems that the python set is not implemented on the Internet. This shows that python users do not seem to care about efficiency issues. On the other hand, according to online documents, python is slower than java. As a c ++ programmer, I used to despise java's operational efficiency. It turns out that python is not as good as java! But java is a virtual machine and python is an interpreter. Why is python slower? The reason is that python is more "Object-Oriented". All types of python are objects, and even the most common integer variables are objects, you must be able to determine the type before you can dynamically create ...... this greatly increases the burden of running, so the running efficiency is so bad. In contrast, if the same program is written in cython and only the variable type is declared, the running efficiency will be improved by 35%.

I have used openmp before. See the previous blog "simple windows multi-threaded program". It seems that openmp is an artifact, which facilitates program writing and can utilize multiple cpu cores to greatly improve the running efficiency. The question is, can openmp be used in python? The answer is pessimistic. The default Implementation of python is cpython, which is implemented by c. Most of the functions of c are not thread-safe, to use these functions to implement and run thread security, python imposes GIL (Global Interpreter Lock) restrictions, that is, at the same time, only one thread can access the python interpreter. However, this makes the concurrency in python very difficult.

However, there are no methods at all. cython now supports openmp. What is cython? What is the relationship with python and cpython? Python is a scripting language, cpython is a python interpreter implemented by c, and cython is another programming language between python and c. In fact, cython's original design is also like this. It must use python's fast programming speed and the running efficiency of C language. A notable difference between cython and python is that all the variables in cython must clearly declare the variable type-this alone, cython runs 35% more efficiently than python in the same program! Although cython is an independent programming language, it seems that you don't need to write programs independently, but use it to compile python c extensions (using c to efficiently implement some programs, call python again ). In the past few days, we tried to use openmp on cython and write the c extension of python to call python.

2. Environment

Windows 7 + 32bit + vistual studio 2008 + python 2.7.3 + eric4 are the default installation paths.

3. Install cython

Download the Cython-0.20.1 on the official website, switch from the console to the path of cython, run setup. py on the way to compile and install it, no other problems encountered.

As you can see on the Internet, many people encounter many problems during installation. Basically, they cannot find the c ++ compiler. The specific manifestation is that they cannot find a prompt called ".... bat files. The solution is to install mingw (gcc version in windows), modify a. cfg file, and specify to build with this compiler.

I have not encountered any problems during the installation process. According to the explanation on the internet, it seems that cpython of python2.7 is compiled using vistual studio 2008. By default, the compiler is correct, so there is no problem. In short, it's okay to install it.

4. Write the pyx File

The pyx file is a python c extension file. The code must comply with cython specifications and can be written in any editor. I wrote it on eric4. As a result, it is interpreted by default using the python interpreter. It also prompts a bug, "syntax error ". Ignore him. Originally, cython syntax is not supported in python. Create the TestOMP. pyx file and write the following code in the file:

from cython.parallel import prange, parallel, threadidfrom libc.stdio cimport printfdef Test():    cdef int i = 0    cdef int sum = 0    for i in prange(1000000, num_threads=2, nogil=True):          printf ("%d\n", i)

The first sentence introduces the parallel processing module in cython, especially prange. In my understanding, prange is "python 'range' of parallel version", which is a parallel loop. The second sentence introduces the 'printf' function in C language. The entire file defines a Test function. We can see that each variable must declare the type before use. In prange, The 'num _ Threads' parameter is used to set the number of concurrent threads. Nogil indicates 'no gil (Global Interpreter Lock) '. To obtain parallelism, this parameter must be set. In the loop process, the c library function printf is called to print each integer.

5. Write the setup. py file.

The above pyx file is only a source code file. To be called by python and run, it is not enough to write the source code. Specifically, it should be converted into a. c or. c ++ file and further converted into a. pyd file. Pyd files are directly usable. To achieve the above purpose, we need to write a setup. py script, as shown below:

#!/usr/bin/python#python version: 2.7.3#Filename: SetupTestOMP.py# Run as:  #    python setup.py build_ext --inplace    import sys  sys.path.insert(0, "..")    from distutils.core import setup  from distutils.extension import Extension  from Cython.Build import cythonize  from Cython.Distutils import build_ext  # ext_module = cythonize("TestOMP.pyx")  ext_module = Extension("TestOMP",            ["TestOMP.pyx"],            extra_compile_args=["/openmp"],            extra_link_args=["/openmp"],            )  setup(    cmdclass = {'build_ext': build_ext},ext_modules = [ext_module], ) 

This is a python script and can be run under the python interpreter. In the console, run the following command 'python setup. py build_ext -- inplace 'to generate the TestOMP. pyd file. Of course, there are also some miscellaneous files, such as the 'lib' file under the 'build 'directory. This indicates that this is in the windows vistual studio environment. In the linux + gcc environment, the. so file is generated, and the "/openmp" option must be written as "-fopenmp"

6. Write the TestOMP. py file

The above two steps are equivalent to writing a python efficiency bottleneck module (which needs to be located using the profile tool before) into the python c extension form with more efficient code. Next, it is to call them in python code. TestOMP. py is the script for this call, as shown below:

from TestOMP import TestTest()

This is very easy, and the import is called. In the console, enter "python TestOMP. py" to run.

7. Results

50756950757050757150757250757350757450757550757650757750757850757991859186918791889189919091919192919391949195919691979198919992009201920292039204920592069207920892099210921192129213921492159216

The above is a clip output on the console. As you can see, prange divides the 1000000 cycle into two intervals: [0, 500000) and (,]. The two loops run in parallel and are Output Using console IO.

8. Corresponding c ++ Program

At the same time, a peer-to-peer c ++ program is written as follows:

# Include
 
  
Using namespace std; void PlayOMP (void) {# pragma omp parallel for num_threads (2) for (int I = 0; I <1000000; I ++) cout <
  
In addition, choose Configuration Properties> C/C ++> Language> OpenMP Support and select Yes from the drop-down menu. And from C: \ Program Files \ Microsoft Visual Studio 9.0 \ VC \ redist \ x86 \ Microsoft. VC90.OPENMP and C: \ Program Files \ Microsoft Visual Studio 9.0 \ VC \ redist \ Debug_NonRedist \ x86 \ Microsoft. copy vcomp90d in the VC90.DebugOpenMP directory respectively. dll and vcomp90.dll files to the current directory of the project file, or set the above two paths to the environment variables.
Compile and run. The result is the same as that in python-but it seems faster!

.
Reprinted please indicate the source: http://blog.csdn.net/xceman1997/article/details/26977483