Tutorials for extending and accelerating Python programs using Pyrex

Pyrex is a language specifically designed to write Python extension modules. According to the Pyrex Web site, "It is designed to build a bridge between the friendly and easy-to-use advanced Python world and the messy low-level C world. "While almost all Python code can be used as a valid Pyrex code, you can add optional static type declarations in your Pyrex code so that these declared objects run at the speed of the C language."
Accelerating Python

In a sense, Pyrex is just one part of the evolving Python Class language series: Jython, IronPython, Prothon, Boo, Vyper (no one now), Stackless Python (in one way), or Parrot runtime (in a different way). In terms of language, Pyrex essentially adds a type declaration to Python. Several of its other changes are not so important (but the expansion of the For loop is pretty).

However, the reason you really want to use Pyrex is that the modules it writes run faster than pure Python and can be much faster.

In fact, Pyrex will generate a C program from the Pyrex code. Intermediate file module.c can still be used for manual processing. However, for "normal" Pyrex users, there is no reason to modify the generated C module. Pyrex itself allows you to access the C-class code that is critical to speed, saving you from writing memory allocations, recycling, pointer operations, function prototypes, and more. Pyrex can also seamlessly handle all the interfaces of Python-level objects, usually by declaring variables as Pyobject structures where necessary, and by using Python C-API calls for memory processing and type conversions.

For the most part, Pyrex doesn't need to keep boxing (box) and unboxing (unbox) operations on simple data type variables, so it's faster than Python. For example, the int type in Python is an object with many methods. It has an inheritance tree and has a calculated "method parsing order (Mothod resolution ORDER,MRO)". It has an allocation and recycling method that can be used for memory processing. It knows when to convert itself to a long type, and how to perform numeric operations on other types of values. All of these additional features mean that more levels of indirection or condition checking are required when working with an int object. On the other hand, the C or Pyrex int variable is only one area in memory where each bit is set to 1 or 0. Using the C/pyrex int type for processing does not involve any indirect operations or condition checks. A CPU "plus" operation can be done in the silicon chip.

In a carefully selected case, the Pyrex module can run 40 to 50 times times faster than the Python version of the same module. However, compared with the modules written in C itself, the Pyrex version of the module is almost no longer than the Python version of the module, the code is more like Python, rather than c.

Of course, when you start talking about accelerated (class) Python modules, Pyrex is not the only tool available. In the Python developer's choice, you can also use Psyco. Psyco can keep the code very brief; it is a JIT Python code compiler (x86) in the machine code. Unlike Pyrex, Psyco does not precisely define the type of a variable, but instead creates several possible machine codes for each Python block of code based on each assumption that the data might be of any kind. If the data is a simple type in a given code snippet, such as int, then this code (which is more pronounced if it is a loop) can run quickly. For example, X can be of type int in a loop that executes 1 million times, but it can still be a value of type float at the end of the loop. Psyco can use the same type as the type explicitly specified in Pyrex to speed up the loop.

Although Pyrex is not difficult, Psyco is easier to use. Using Psyco is just adding a few lines to the end of the module; in fact, if you add the correct code, the module can run the same way even when the Psyco is not available (just slower).
Listing 1. Use Psyco only if Psyco is available

# import Psyco If availabletry:  import Psyco  psyco.full () except Importerror:  Pass

To use Pyrex, you need to modify the code more (but only a little more), the system also needs to install a C compiler, and correctly configure the system to generate the Pyrex module. Although you can distribute binary Pyrex modules, the Python version, architecture, and optimization options required by the end user must match in order for your module to run elsewhere.

Speed First Experience

I recently created a pure Python Hashcash implementation for DeveloperWorks's article Beat spam using Hashcash, but basically hashcash is a technique that uses SHA-1 to provide CPU work. Python has a standard module sha, which makes writing hashcash very simple.

Unlike the 95% Python program I wrote, the slow speed of the Hashcash module upset me, at least a little bit. By design, this protocol is to eat up all of the CPU cycles, so the efficiency of operation is critical. HASHCASH.C's ANSI C binaries run at 10 times times the speed of this hashcash.py script. And the speed of the optimized hashcash.c binaries with Ppc/altivec enabled is 4 times times the normal ANSI C version (1Ghz G4/altivec at 3Ghz when processing Hashcash/sha operations) 4?/mmx;g5 speed will be faster). So the tests on my tipowerbook show that this module is 40 times times slower than the optimized C version (though the gap on x86 is not so large).

Because this module is running slowly, Pyrex may be a good acceleration method. At least I think so. The first thing about the "Pyrex" hashcash.py (after installing Pyrex, of course) is simply to copy it to Hashcash_pyx.pyx and try to handle it like this:

$ pyrexc Hashcash_pyx.pyx

Creating a binary module

Running this command generates a HASHCASH.C file (which makes some minor changes to the source file). Unfortunately, adjusting the GCC switches just right for my platform requires a bit of skill, so I decided to use the recommended shortcut to let Distutils do some work for me. The standard Python installation knows how to use the local C compiler during module installation and how to use distutils to simplify the sharing of Pyrex modules. I created a setup_hashcash.py script that looks like this:
Listing 2. setup_hashcash.py Script

From Distutils.core import setupfrom distutils.extension import extensionfrom pyrex.distutils import Build_extsetup ( Name = "Hashcash_pyx", ext_modules=[  Extension ("Hashcash_pyx", ["Hashcash_pyx.pyx"], libraries = [])  ], Cmdclass = {' Build_ext ': Build_ext})

Run the following command to fully compile a C-based extension module Hashcash:

$ python2.3 prime_setup.py Build_ext--inplace

Code modification

I have simplified the work of generating C-based modules from Hashcash.pyx. In fact, I need to make two changes to the source code and find the location to be modified by finding the location where the Pyrexc complained. In the code, I used an unsupported list and put it in a normal for loop. This is very simple. I also modified the increment assignment from Counter+=1 to counter=counter+1.

That's all. This is my first Pyrex module.

Test speed

In order to be able to simply test the speed of the module being developed, I wrote a simple test program to run different versions of the module:
Listing 3. Test procedure hashcash_test.py

#!/usr/bin/env python2.3import time, sys, optparsehashcash = __import__ (sys.argv[1]) start = Time.time () print Hashcash.mint (' mertz@gnosis.cx ', bits=20) timer = Time.time ()-startsys.stderr.write ("%0.4f seconds (%d hashes per Second) \ n "%    (timer, Hashcash.tries[0]/timer))

The exciting point is that I decided to look at how the speed can be improved only by compiling Pyrex. Note that in all of the examples below, the real time varies greatly and is random. What we want to see is "hashes per second", which can measure speed accurately and reliably. So compare the pure Python and Pyrex:
Listing 4. Comparison of pure Python and "pure Pyrex"

$./hashcash_test.py hashcash1:20:041003:mertz@gnosis.cx::i+lynupv:167dca13.7879 seconds (106904 hashes per second) $. /hashcash_test.py Hashcash_pyx >/dev/null6.0695 seconds (89239 hashes per second)

Oh! The use of Pyrex is almost 20% slower. That's not what I expected. Now it's time to analyze where the code might be accelerating. The following short function will attempt to consume all the time:
Listing 5. Functions in the hashcash.py

def _mint (Challenge, bits):  "Answer a ' generalized Hashcash ' challenge '"  counter = 0  hex_digits = Int (Ceil ( Bits/4.))  Zeros = ' 0 ' *hex_digits  hash = sha while  1:    digest = hash (Challenge+hex (counter) [2:]). Hexdigest ()    if Digest[:hex_digits] = = Zeros:      tries[0] = counter      return Hex (counter) [2:]    counter + = 1

I need to use the advantages of the Pyrex variable declaration to accelerate. Some variables are obviously integers, others are obviously strings--we can specify these types. I will use the improved for loop for Pyrex when making modifications:
Listing 6. Mint function with minimal Pyrex improvement

Cdef _mint (challenge, int bits):  # Answer A ' Generalized Hashcash ' challenge ' "  cdef int counter, hex_digits, i
  
   cdef char *digest  hex_digits = Int (ceil (BITS/4.))  hash = Sha for  counter from 0 <= counter < sys.maxint:    py_digest = hash (Challenge+hex (counter) [2:]). hexdige St ()    Digest = Py_digest for    i from 0 <= i < hex_digits:      if digest[i]! = C ' 0 ': Break    else:      trie S[0] = counter      return Hex (counter) [2:]
  

So far everything is very simple. I only declare some variable types that I already know and use the cleanest Pyrex counter loops. A small trick is to assign Py_digest (a Python string) to digest (a C/pyrex string) to determine its type. After the experiment, I also found that the loop string comparison operation is very fast. What benefits will this bring?
Listing 7. The speed result of Pyrex mint function

$./hashcash_test.py hashcash_pyx2 >/dev/null20.3749 seconds (116636 hashes per second)

It's much better now. I've made some minor improvements to the original Python, which can slightly improve the speed of the original Pyrex module. But the effect is not obvious, only to raise a very small percentage.
Analysis

Some things seem wrong. A few percent increase in speed and the Pyrex home page (and many Pyrex users) up to 40 times times as much as there is a big gap. Now it's time to take a look at where this Python _mint () function really consumes. There is a quick script (not given here) that can decompose the complex operations of SHA (Challenge+hex (counter) [2:]). Hexdigest ():
Listing 8. Time consumption of Hashcash's mint function

1000000 empty loops:   0.559------------------------------1000000 sha () s:     2.3321000000 hex () [2:]s:   3.151  just Hex () s:     <2.471>1000000 concatenations:0.8551000000 hexdigest () s:  3.742------------------- -----------Total:         10.079

Obviously, I'm not able to remove this loop from the _mint () function. Although the Pyrex improved for loop may have a little speedup, the entire function is primarily a loop. I can't remove the call to Sha () unless I'm using Pyrex to re-implement SHA-1 (even if I do, I'm not confident that I can do better than the author of the Python Standard SHA module). And if I want to get an SHA. The hash value of the SHA object can only be called. Hexdigest () or. Digest (); the former is faster.

The real solution now is the conversion of the Hex () to the counter variable, and the consumption of the time slice in the result. I might need to use the PYREX/C string connection operation instead of the Python string object. However, the only way I've ever seen to avoid hex () conversions is to manually build a suffix outside of the nested loop. Although this avoids the conversion of int to char type, you need to generate more code:
Listing 9. Fully Pyrex-optimized mint function

Cdef _mint (char *challenge, int bits):  cdef int hex_digits, I0, I1, I2, i3, I4, i5  cdef Char *ab, *digest, *trial, *suffix  suffix = ' ****** '  ab = alphabet  hex_digits = Int (ceil (BITS/4.))  hash = Sha for  i0 from 0 <= I0 < in:    suffix[0] = ab[i0] for    i1 from 0 <= i1 <:      Suffix[1] = AB[I1] for      i2 from 0 <= i2 < in:        suffix[2] = Ab[i2] for        i3 from 0 <= i3 <:          Suffix[3] = AB[I3] for          I4 from 0 <= I4 < in:            suffix[4] = Ab[i4] for            i5 from 0 <= i5 <:              Suffix[5] = AB[I5]              py_digest = hash (challenge+suffix). Hexdigest ()              digest = Py_digest for              i from 0 <= i < hex_dig Its:                if digest[i]! = C ' 0 ': Break              else:                return suffix

Although this Pyrex function still looks simpler and easier to read than the corresponding C function, it is actually more complex than the original pure Python version. In this way, expanding the suffix generation in pure Python has some negative effects on the overall speed compared to the original version. In Pyrex, as you would expect, these nested loops are seldom time consuming, so I save the cost of conversion and tick scheduling:
Listing 10. The speed result after Pyrex optimization of mint function

$./hashcash_test.py hashcash_pyx3 >/dev/null13.2270 seconds (166125 hashes per second)

Of course, it's much better than when I started. But the speed is just twice times higher. The problem with most of the time is that (and here too) it consumes too much time on the call to the Python library, and I can't write code to improve the speed of these calls.
Disappointing comparisons

The speed increase of 50% to 60% seems to be worthwhile. I didn't write much code to achieve this goal. However, if you think that you added two statements to the original Python version of import Psyco;psyco.bind (_mint), then this acceleration method will not give you a deep impression:
Listing 11. Accelerating results of Psyco of mint function

$./hashcash_test.py Hashcash_psyco >/dev/null15.2300 seconds (157550 hashes per second)

In other words, Psyco adds two lines of common code, almost achieving the same goal. Of course, Psyco can only be used on x86 platforms, and Pyrex may be executed on all environments that have a C compiler. But for this particular example, Os.popen (' hashcash-m ' +options) will be much faster than Pyrex and Psyco (assuming, of course, that you can use the C tool Hashcash).

