Tutorial on using Pyrex to extend and accelerate Python programs _python

Pyrex is a language specifically designed to write Python extension modules. According to the Pyrex Web site, "It is designed to build a bridge between the friendly and easy-to-use high-level Python world and the messy, low-level C world." "Although almost all Python code can be used as a valid Pyrex code, you can add optional static type declarations to the Pyrex code to make these declared objects run at the speed of C."
Accelerate Python

In a sense, Pyrex is just one part of the evolving Python language series: Jython, IronPython, Prothon, Boo, Vyper (no one is using it now), Stackless Python (in one way) or Parrot runtime (in a different way). In terms of language, Pyrex essentially adds a type declaration in Python. Some of its other changes are less important (though the extension to the For loop is beautiful).

However, the reason you really want to use Pyrex is that it writes a module that runs faster than pure Python and can be a lot quicker.

In fact, Pyrex will generate a C program from the Pyrex code. Intermediate file module.c can still be used for manual processing. However, for "ordinary" Pyrex users, there is no reason to modify the generated C module. Pyrex itself allows you to access those C-level code that is critical to speed, saving you the effort to write memory allocations, recycling, pointer operations, function prototypes, and more. Pyrex can also seamlessly handle all the interfaces of a Python-level object, usually by declaring variables as Pyobject structures where necessary and using Python C-API calls for memory processing and type conversions.

For the most part, Pyrex does not need to keep boxing (box) and unboxing (unbox) operations on simple data type variables, so it is faster than Python. For example, the int type in Python is an object with many methods. It has an inheritance tree and has a computed "method parsing order (Mothod resolution ORDER,MRO)". It has allocation and recycling methods that can be used for memory processing. It knows when to convert itself to a long type, and how to perform numeric operations on other types of values. All of these additional features mean that more level of indirection or conditional checking is required for processing with an int object. On the other hand, the int variable of C or Pyrex is only one region in memory with each bit set to 1 or 0. Processing with the int type of C/pyrex does not involve any indirect operations or condition checking. A CPU "plus" operation in the silicon chip can be done.

In a carefully selected case, the Pyrex module can run 40 to 50 times times faster than the Python version of the same module. But compared to modules written with C itself, the Pyrex version of the module is almost never longer than the Python version of the module, and the code is more like Python than C.

Of course, when you start talking about accelerating (class) Python modules, Pyrex is not the only tool available. In the Python developer's choice, you can also use Psyco. Psyco can keep the code very short; it is a JIT Python code compiler in the (x86) machine code. Unlike Pyrex, Psyco does not precisely qualify the type of the variable, but instead creates several possible machine codes for each Python code block based on each assumption that the data might be of that type. If the data is a simple type, such as int, in a given code snippet, the code (which is more pronounced if it is a loop) can be run quickly. For example, x can be an int type in a loop that executes 1 million times, but it can still be a float at the end of the loop. Psyco can accelerate loops using the same type as the type explicitly specified in Pyrex.

Although Pyrex is also not difficult, but Psyco is more simple and easy to use. Using Psyco is just adding a few lines to the end of the module; in fact, if you add the correct code, the module can run (albeit slowly) even when Psyco is not available.
Listing 1. Use Psyco only when Psyco is available

# import Psyco if available
try:
  import Psyco
  psyco.full ()
except:
  Pass

To use Pyrex, you need to make more changes to your code (but only a little more), and you will need to install a C compiler and properly configure the system that generates the Pyrex module. Although you can distribute binary Pyrex modules, the Python version, architecture, and end-user needs optimization options must match in order to make your module run elsewhere.

Initial speed Experience

I recently created a pure Python Hashcash implementation for DeveloperWorks's article Beat spam using Hashcash, but basically, Hashcash is a technology that uses SHA-1 to provide CPU work. Python has a standard module sha, which makes writing hashcash very simple.

Unlike the 95% Python program I wrote, the slow speed of the Hashcash module bothers me, at least for a little bit. According to the design, the protocol is to eat up all the CPU cycle, so running efficiency is very critical. The hashcash.c ANSI C binaries run at 10 times times the speed of this hashcash.py script. And the speed of the Ppc/altivec optimized hashcash.c binaries is 4 times times that of the normal ANSI C version (1Ghz G4/altivec is the equivalent of 3Ghz when processing Hashcash/sha operations Pentium The speed of 4?/mmx;g5 will be faster). As a result, the tests on my Tipowerbook show that the module is 40 times times slower than the optimized C version (although the gap between the x86 is not so large).

Because this module is running slowly, it is possible that Pyrex will be a better method of acceleration. At least I think so. The first thing to "Pyrex" hashcash.py (after installing Pyrex, of course) is simply to copy it as Hashcash_pyx.pyx and try to do this:

$ pyrexc Hashcash_pyx.pyx

To create a binary module

Running this command generates a HASHCASH.C file (which makes some minor changes to the source file). Unfortunately, adjusting the GCC switch just right for my platform requires some skill, so I decided to take the recommended shortcut and let Distutils do some work for me. The standard Python installation knows how to use the local C compiler during module installation, and how to use distutils to simplify the sharing of Pyrex modules. I created a setup_hashcash.py script, as follows:
Listing 2. setup_hashcash.py Script

From Distutils.core Import Setup from
distutils.extension import extension to
pyrex.distutils import build_ Ext
Setup (
 name = "Hashcash_pyx",
 ext_modules=[
  Extension ("Hashcash_pyx", ["Hashcash_pyx.pyx"], libraries = [])
  ],
 cmdclass = {' Build_ext ': Build_ext}
)

Run the following command to fully compile a C-based extension module Hashcash:

$ python2.3 prime_setup.py Build_ext--inplace

Code modification

I have simplified the work of generating C-based modules from Hashcash.pyx. In fact, I need to make two modifications to the source code and find the location to be modified by looking for the location where Pyrexc complained. In the code, I used an unsupported list and put it into a normal for loop. It's very simple. I also modified the incremental assignment from Counter+=1 to counter=counter+1.

So much. This is my first Pyrex module.

Test speed

To simply test the speed of the module to be developed, I wrote a simple test program to run different versions of the module:
Listing 3. Test program hashcash_test.py

#!/usr/bin/env python2.3
import time, sys, optparse
hashcash = __import__ (sys.argv[1))
start = Time.time ()
print hashcash.mint (' mertz@gnosis.cx ', bits=20)
timer = time.time ()-start
sys.stderr.write ("%0.4f Seconds (%d hashes per second) \ n "%
    (timer, Hashcash.tries[0]/timer))

The exciting point is that I decided to take a look at how to improve speed only through Pyrex compilation. Note that in all of the following examples, real time changes are very large and are random. What we want to see is "hashes per second", which can measure speed accurately and reliably. So compare the sheer Python and Pyrex:
Listing 4. A comparison between pure Python and "pure Pyrex"

$./hashcash_test.py Hashcash
1:20:041003:mertz@gnosis.cx::i+lynupv:167dca
13.7879 seconds (106904 hashes Per second)
$./hashcash_test.py hashcash_pyx >/dev/null
6.0695 seconds (89239 hashes per second)

Oh! The use of Pyrex is almost 20% slower. That's not what I expected. Now it's time to analyze where the code might be speeding up. The following short function will attempt to consume all of the time:
Listing 5. Functions in the hashcash.py

def _mint (Challenge, bits):
  "Answer a ' generalized Hashcash ' challenge '"
  counter = 0
  hex_digits = Int (Ceil ( Bits/4.))
  Zeros = ' 0 ' *hex_digits
  hash = sha while
  1:
    digest = hash (Challenge+hex (counter) [2:]). Hexdigest ()
    if Digest[:hex_digits] = = Zeros:
      tries[0] = counter return
      hex (counter) [2:]
    counter + 1

I need to use the advantages of the Pyrex variable declaration to accelerate. Some variables are obviously integers, while others are obviously strings--we can specify these types. When I make a change, I will use the improved for loop for Pyrex:
Listing 6. An improved mint function with minimal Pyrex

Cdef _mint (challenge, int bits):
  # Answer A ' Generalized Hashcash ' challenge ' "cdef
  int counter, hex_digits, i
   
    cdef char *digest
  hex_digits = Int (ceil (BITS/4.))
  hash = Sha for
  counter from 0 <= counter < sys.maxint:
    py_digest = hash (Challenge+hex (counter) [2:]). hexdige St ()
    Digest = Py_digest for
    i-0 <= i < hex_digits:
      if digest[i]!= C ' 0 ': Break
    else:
      Tri Es[0] = counter return
      hex (counter) [2:]


   

So far everything is very simple. I only declare some variable types that I already know and use the cleanest Pyrex counter loops. One trick is to assign Py_digest (a Python string) to digest (a C/pyrex string) to determine its type. After the experiment, I also found that the loop string comparison operation speed is very fast. What benefits will this bring?
Listing 7. The speed result of Pyrex mint function

$./hashcash_test.py hashcash_pyx2 >/dev/null
20.3749 seconds (116636 hashes per second)

It's much better now. I've made some minor improvements to the original Python, which can slightly improve the speed of the original Pyrex module. But the effect is not obvious, only a small percentage increase.
Analysis

Some things seem wrong. A few percent increase in speed has a big gap with the Pyrex home page (and many Pyrex users) up to 40 times times. Now it's time to take a look at where the Python _mint () function really consumes. There is a quick script (not shown here) that can decompose complex operations Sha (Challenge+hex (counter) [2:]). Hexdigest ():
Listing 8. Time consumption of the mint function of Hashcash

1000000 empty loops:   0.559
------------------------------
1000000 sha () s:     2.332 1000000 hex
() [ 2:]s:   3.151
  just Hex () s:     <2.471> 1000000 concatenations:0.855 1000000 hexdigest
() s:  3.742
------------------------------total
:         10.079

Obviously, I'm not able to remove this loop from the _mint () function. Although the Pyrex for the improved for loop may be a little accelerated, the entire function is primarily a loop. Nor can I remove the call to Sha () unless I want to use Pyrex to implement SHA-1 (even if I do, I am not confident that I can do better than the author of the Python Standard SHA module). And if I want to get an SHA. The hash value of the SHA object, you can only call. Hexdigest () or. Digest (), the former faster.

What is really going to be solved now is the conversion of Hex () to the counter variable and the consumption of the time slice in the result. I might need to use pyrex/c string concatenation instead of a Python string object. However, the only way I've ever seen to avoid hex () conversion is by manually building a suffix outside of the nested loop. While this avoids the int to char type conversion, more code needs to be generated:
Listing 9. Fully Pyrex optimized mint function

 Cdef _mint (char *challenge, int bits): Cdef int hex_digits, I0, I1, I2, i3, I4, i5 CDE
  F Char *ab, *digest, *trial, *suffix suffix = ' hu Jintao ' ab = alphabet hex_digits = Int (ceil (BITS/4.)) hash = Sha for i0 from 0 <= i0 < 55:suffix[0] = Ab[i0] for i1 from 0 <= i1 < 55:suffix[1] =
          AB[I1] for i2 from 0 <= i2 < 55:suffix[2] = Ab[i2] for i3 from 0 <= i3 < 55: SUFFIX[3] = Ab[i3] for i4 from 0 <= i4 < 55:suffix[4] = Ab[i4] for i5 from 0 <
              = i5 < 55:suffix[5] = Ab[i5] Py_digest = hash (challenge+suffix). Hexdigest ()
              Digest = py_digest for i-0 <= i < hex_digits:if digest[i]!= C ' 0 ': Break Else:return suffix 

Although the Pyrex function still looks more readable than the corresponding C function, it is actually more complex than the original pure Python version. In this way, expanding the suffix generation in pure Python has a negative impact on the overall speed compared to the original version. In Pyrex, as you would expect, these nested loops are very time-consuming, so I save the cost of conversion and time-sharing:
Listing 10. The speed result of mint function Pyrex optimization

$./hashcash_test.py hashcash_pyx3 >/dev/null
13.2270 seconds (166125 hashes per second)

Of course, this is a lot better than when I started. But the speed is only twice times higher. The problem with most of the time (and here) is that too much time is spent on calls to the Python library, and I'm not able to code these calls to improve speed.
Disappointing comparisons

A speed increase of 50% to 60% seems to be worthwhile. I didn't write much code to achieve this goal. But if you think you added two statements to import Psyco;psyco.bind (_mint) in the original Python version, this acceleration method won't impress you:
Listing 11. Accelerated results of mint function Psyco

$./hashcash_test.py Hashcash_psyco >/dev/null
15.2300 seconds (157550 hashes per second)

In other words, Psyco has added two lines of common code, almost achieving the same goal. Of course, Psyco can only be used for x86 platforms, while Pyrex is executed on all environments with a C compiler. But for this particular example, Os.popen (' hashcash-m ' +options) is much faster than Pyrex and Psyco (assuming, of course, that C tool Hashcash is available).