Tutorial on using Pyrex to expand and accelerate Python programs

This article mainly introduces how to use Pyrex to expand and accelerate Python programs, from the IBM official technical documentation, for more information about how to compile Python extension modules, see Pyrex. According to the Pyrex Web site, "it is designed to build a bridge between the friendly and easy-to-use advanced Python world and the messy Low-Level C world ." Although almost all Python code can be used as valid Pyrex code, you can add optional static type declarations in Pyrex code, so that these declared objects run at the speed of the C language.
Accelerate Python

In a sense, Pyrex is only part of the ever-evolving Python class language series: Jython, IronPython, Prothon, Boo, and Vyper (no one is using it now), Stackless Python (in one way) or Parrot runtime (in another way ). In terms of language, Pyrex essentially adds a type declaration to Python. Its other changes are not so important (but the expansion of the for loop is pretty good ).

However, the reason you really want to use Pyrex is that the modules written by Pyrex run faster and may be much faster than pure Python.

In fact, Pyrex generates a C program from the Pyrex code. The intermediate file module. c can still be used for manual processing. However, for "common" Pyrex users, there is no reason to modify the generated C module. Pyrex allows you to access the C-level code that is critical to speed, and saves you the effort to write memory allocation, collection, Pointer operations, and function prototypes. Pyrex can also seamlessly process all interfaces of Python-level objects; usually it is implemented by declaring variables as PyObject structures where necessary and using Python C-API calls for memory processing and type conversion.

In most cases, Pyrex does not need to perform box and unbox operations on simple data type variables, so it is faster than Python. For example, the int type in Python is an object with many methods. It has an inheritance tree and has a computed "method resolution order (MRO )". It can be allocated and recycled for memory processing. It knows when to convert itself to a long type, and how to perform numerical operations on values of other types. All of these additional functions mean that more levels of indirect processing or condition checks are required when int objects are used for processing. On the other hand, the int variable of C or Pyrex is only a region where each bit in the memory is set to 1 or 0. Using the int type of C/Pyrex for processing does not require any indirect operations or condition checks. A cpu "add" operation can be completed in the silicon chip.

In case of careful selection, the Pyrex module can run 40 to 50 times faster than the same module of Python version. However, compared with modules written in C, the modules in Pyrex version are almost no longer than those in Python version, and the code is more similar to Python than C.

Of course, when you start talking about the acceleration (class) Python module, Pyrex is not the only available tool. You can also use Psyco in the selection of Python developers. Psyco can keep the code very short; it is a JIT Python code compiler in (x86) machine code. Unlike Pyrex, Psyco does not precisely limit the type of a variable. Instead, it creates several possible machine codes for each Python code block based on each assumption of the data type. If the data in a given code segment is a simple type, such as int, this code (if it is a loop, this situation is more prominent) can run quickly. For example, x can be an int type in a 1 million-time loop, but it can still be a float value at the end of the loop. Psyco can accelerate the loop by using the same type explicitly specified in Pyrex.

Although Pyrex is not difficult, Psyco is easier to use. Psyco is used only by adding a few lines at the end of the module. In fact, if the correct code is added, the module can run the same way even when Psyco is unavailable (but the speed is slow ).
Listing 1. Psyco is used only when Psyco is available

# Import Psyco if availabletry:  import psyco  psyco.full()except ImportError:  pass

To use Pyrex, you need to modify the code more (but only a little more) and install a C compiler in the system, correctly configure the system that generates the Pyrex module. Although you can distribute binary Pyrex modules, the Python version, architecture, and optimization options required by end users must match to make your modules run elsewhere.

Initial speed experience

I recently created a Python-only hashcash implementation for Beat spam using hashcash, a developerWorks article. but basically, hashcash is a technology that uses SHA-1 to provide CPU work. Python has a standard module sha, which makes it very easy to write hashcash.

Unlike the 95% Python program I wrote, the slow speed of the hashcash module upset me at least a little bit. According to the design, this protocol is to eat up all the CPU cycles, so the operation efficiency is critical. The ANSI c binary file of hashcash. C runs 10 times faster than the hashcash. py script. The hashcash after PPC/Altivec optimization is enabled. the speed of the c binary file is four times that of the normal ansi c version (1 Ghz G4/Altivec is equivalent to 3 Ghz Pentium4 when processing hashcash/SHA operations? /MMX; G5 is faster ). Therefore, tests on my TiPowerbook show that the speed of this module is 40 times slower than that of the optimized C version (but the gap in x86 is not that big ).

Because this module runs slowly, Pyrex may be a good acceleration method. At least I think so. The first thing about "Pyrex" hashcash. py (after installing Pyrex, of course) is to simply copy it as hashcash_pyx.pyx and try to handle it like this:

$ pyrexc hashcash_pyx.pyx

Create a binary module

Run this command to generate a hashcash. c File (this will slightly change the source file ). Unfortunately, adjusting the gcc switch just fits my platform and requires some skills, so I decided to adopt the recommended shortcut so that distutils could do some work for me. Standard Python installation knows how to use a local C compiler during module installation and how to use distutils to simplify Pyrex module sharing. I created a setup_hashcash.py script, as shown below:
Listing 2. setup_hashcash.py script

from distutils.core import setupfrom distutils.extension import Extensionfrom Pyrex.Distutils import build_extsetup( name = "hashcash_pyx", ext_modules=[  Extension("hashcash_pyx", ["hashcash_pyx.pyx"], libraries = [])  ], cmdclass = {'build_ext': build_ext})

Run the following command to compile a C-based extension module hashcash:

$ python2.3 prime_setup.py build_ext --inplace

Code modification

I simplified the process of generating C-based modules from hashcash. pyx. In fact, I need to modify the source code in two ways; find the location where pyrexc complained to find the location to be modified. In the code, I used an unsupported list and put it into a common for loop. This is very simple. I also changed the incremental value from counter + = 1 to counter = counter + 1.

That's all. This is my first Pyrex module.

Testing speed

In order to test the speed improvement of the modules to be developed, I wrote a simple test program to run modules of different versions:
Listing 3. test program hashcash_test.py

#!/usr/bin/env python2.3import time, sys, optparsehashcash = __import__(sys.argv[1])start = time.time()print hashcash.mint('mertz@gnosis.cx', bits=20)timer = time.time()-startsys.stderr.write("%0.4f seconds (%d hashes per second)\n" %    (timer, hashcash.tries[0]/timer))

The excitement is that I decided to see how the speed can be improved only through Pyrex compilation. Note that in all the examples below, real time changes greatly and are random. We should look at "hashes per second", which can accurately and reliably measure the speed. Therefore, compare the pure Python and Pyrex:
Listing 4. Comparison between pure Python and pure Pyrex

$ ./hashcash_test.py hashcash1:20:041003:mertz@gnosis.cx::I+lyNUpV:167dca13.7879 seconds (106904 hashes per second)$ ./hashcash_test.py hashcash_pyx > /dev/null6.0695 seconds (89239 hashes per second)

Oh! Using Pyrex is almost 20% slower. This is not what I expected. Now we should analyze the possible code acceleration points. The following short function will try to consume all the time:
Listing 5. Functions in hashcash. py

def _mint(challenge, bits):  "Answer a 'generalized hashcash' challenge'"  counter = 0  hex_digits = int(ceil(bits/4.))  zeros = '0'*hex_digits  hash = sha  while 1:    digest = hash(challenge+hex(counter)[2:]).hexdigest()    if digest[:hex_digits] == zeros:      tries[0] = counter      return hex(counter)[2:]    counter += 1

I need to use the advantages of Pyrex variable declaration for acceleration. Some variables are obviously integers, while others are obviously strings-we can specify these types. During the modification, I will use the improved for loop of Pyrex:
Listing 6. mint functions improved with minimum Pyrex

cdef _mint(challenge, int bits):  # Answer a 'generalized hashcash' challenge'"  cdef int counter, hex_digits, i  cdef char *digest  hex_digits = int(ceil(bits/4.))  hash = sha  for counter from 0 <= counter < sys.maxint:    py_digest = hash(challenge+hex(counter)[2:]).hexdigest()    digest = py_digest    for i from 0 <= i < hex_digits:      if digest[i] != c'0': break    else:      tries[0] = counter      return hex(counter)[2:]

So far, everything has been very simple. I only declare some variable types that I already know and use the cleanest Pyrex counter loop. One trick is to assign the py_digest (a Python string) value to digest (a C/Pyrex string) to determine its type. After experiments, I also found that Loop strings are faster than each other. What benefits will these bring?
Listing 7. Pyrex mint function speed results

$ ./hashcash_test.py hashcash_pyx2 >/dev/null20.3749 seconds (116636 hashes per second)

This is much better. I have made some minor improvements to the original Python, which can slightly increase the speed of the original Pyrex module. However, the effect is not obvious, but only a small percentage is increased.
Analysis

Something seems wrong. The increase in speed is significantly different from the 40-fold increase on the Pyrex homepage (and many Pyrex users. Now let's take a look at the places in this Python _ mint () function that actually consumed time. There is a quick script (not provided here) that can be used to break down complex operations sha (challenge + hex (counter) [2:]). hexdigest ():
Listing 8. time consumption of the hashcash mint function

1000000 empty loops:   0.559------------------------------1000000 sha()s:     2.3321000000 hex()[2:]s:   3.151  just hex()s:     <2.471>1000000 concatenations: 0.8551000000 hexdigest()s:  3.742------------------------------Total:         10.079

Obviously, I cannot delete this loop from the _ mint () function. Although the for loop improved by Pyrex may accelerate a little, the entire function is mainly a loop. I cannot delete the sha () call, unless I want to use Pyrex to implement SHA-1 again (even if I want to do this, I am not confident that I can do better than the author of The Python Standard sha module ). In addition, if I want to get the hash value of a sha. SHA object, I can only call. hexdigest () or. digest (); the former is faster.

What we need to solve now is the hex () transformation to the counter variable and the consumption of time slices in the result. I may need to use a Pyrex/C string connection operation instead of a Python string object. However, the only way I have seen to avoid hex () conversion is to manually build a suffix out of nested loops. Although this can avoid int to char type conversion, more code needs to be generated:
Listing 9. mint functions fully optimized by Pyrex

cdef _mint(char *challenge, int bits):  cdef int hex_digits, i0, i1, i2, i3, i4, i5  cdef char *ab, *digest, *trial, *suffix  suffix = '******'  ab = alphabet  hex_digits = int(ceil(bits/4.))  hash = sha  for i0 from 0 <= i0 < 55:    suffix[0] = ab[i0]    for i1 from 0 <= i1 < 55:      suffix[1] = ab[i1]      for i2 from 0 <= i2 < 55:        suffix[2] = ab[i2]        for i3 from 0 <= i3 < 55:          suffix[3] = ab[i3]          for i4 from 0 <= i4 < 55:            suffix[4] = ab[i4]            for i5 from 0 <= i5 < 55:              suffix[5] = ab[i5]              py_digest = hash(challenge+suffix).hexdigest()              digest = py_digest              for i from 0 <= i < hex_digits:                if digest[i] != c'0': break              else:                return suffix

Although this Pyrex function seems easier to read than the corresponding C function, it is actually more complex in the original pure Python version. In this way, extension generation in pure Python has a negative effect on the overall speed compared with the original version. In Pyrex, as you expected, these nested loops are rarely time-consuming, saving me the cost of conversion and time-sharing scheduling:
Listing 10. mint function Pyrex optimized speed results

$ ./hashcash_test.py hashcash_pyx3 >/dev/null13.2270 seconds (166125 hashes per second)

Of course, this is much better than when I started. However, the speed improvement is only twice. Most of the time the problem is that (this is also the case) consumes too much time to call the Python library, and I cannot write code for these calls to speed up.
Disappointing comparison

It seems worthwhile to increase the speed by 50% to 60%. I didn't write much code to achieve this goal. However, if you add two statements import psyco; psyco. bind (_ mint) to the original Python version, this acceleration method will not give you a deep impression:
Listing 11. acceleration results of the mint function Psyco

$ ./hashcash_test.py hashcash_psyco >/dev/null15.2300 seconds (157550 hashes per second)

In other words, Psyco can achieve almost the same goal by adding two lines of common code. Of course, Psyco can only be used on the x86 platform, and Pyrex can be executed in all environments with a C compiler. But for this particular example, OS. the speed of popen ('hashcash-M' + options) is much faster than that of Pyrex and Psyco (of course, assuming the C tool hashcash can be used ).