Python Extended Technology Summary __python

In general, all the code that can be consolidated or imported into other Python scripts can be called extensions. You can write extensions in pure python, or you can write extensions in a compiled language such as C + +, and you can even write Python extensions with java,c. A big feature of Python is that the interaction between extensions and interpreters is the same as the common Python module, and the Python module import mechanism is abstract to the extent that the code that uses the module does not understand the specifics of the implementation of the module.

The main reasons for the expansion of Python are three points: (1) Add functionality that is not provided at the core of the additional Python language (2) to improve performance bottlenecks (3) maintain proprietary source code privacy, transfer part of the code from Python to the compiled language to keep proprietary source code private

Method one: Using C API for C extension

This method is the most basic, including three steps: 1. Create application code 2. Use boilerplate to wrap code 3. Compiling
1. Create a extest.c file containing two C functions

Extest.c

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define BUFSIZE ten

int FAC (int n) {
    if (n < 2) return
        1;
    return n * FAC (N-1);
}

string inversion
char *reverse (char *s) {
    Register char t;
    char *p = s;
    Char *q = (s + (strlen (s)-1));
    while (P < q) {
        t = *p;
        *p++ = *q;
        *q--= t;
    }
    return s;
}
<strong><span style= "FONT-SIZE:18PX;" >
</span></strong>

2. Packing your code with a sample

The use of templates is divided into 4 steps:

1. Add a Python header file

2. Add a wrapper function that is shaped like pyobject* module_func () for each function of each module

3. Add an array of shapes like Pymethoddef modulemethods[for each module

4. Increase module initialization function void Initmodule ()

The second step requires some skill, and you need to add a static function for all functions that want to be accessed by the Python environment. The return value type of the function is pyobject*, and in the Python C language extension interface, most functions have one or more parameters of the Pyobject pointer type, and the return value is mostly pyobject pointers. The use of a wrapper function is to pass the value of Python to C and then call the C function to convert the result of the function into a Python object, and then return it to the Python

Step three each array contains a function information, including the name of the function python, the name of the corresponding wrapper function, and a Meth_varargs constant (indicating that the parameter is passed in as a tuple)

Extest.c

#include <stdio.h> #include <stdlib.h> #include <string.h> #include "Python.h"//containing Python header files #defin
    e bufsize int FAC (int n) {if (n < 2) return 1;
return n * FAC (N-1);
    } Char *reverse (char *s) {Register char t;
    char *p = s;
    Char *q = (s + (strlen (s)-1));
        while (P < q) {t = *p;
        *p++ = *q;
    *q--= t;
return s;
    The wrapper function of the//FAC function is static Pyobject * EXTEST_FAC (pyobject *self, Pyobject *args) {int num; if (!) (
    Pyarg_parsetuple (args, "I", &num)) {return NULL;
Return (Pyobject *) py_buildvalue ("I", FAC (num));
    The wrapper function of the//reverse function is static Pyobject * Extest_doppel (pyobject *self, Pyobject *args) {char *orignal;
    Char *reversed;
    Pyobject * retval; if (!) (
    Pyarg_parsetuple (args, "s", &orignal)) {return NULL;
    } retval = (Pyobject *) Py_buildvalue ("ss", Orignal, Reversed=reverse (StrDup (orignal));
    Free (reversed);
return retval;///To create an array of function information for the module static pymethoddef extestmethods[] = {{"FAC", EXTEST_FAC, Meth_varargs}, {"Doppel", Extest_dop
PEL, Meth_varargs},};
 Increase module initialization function void Initextest () {py_initmodule ("extest", Extestmethods);}

In the compile phase, you need to create a setup.py that compiles and links the code through setup.py. Once this step is complete, you can import the extended module directly. You need to import the Distutils package in setup.py. First you create a extension instance for each extension, and then the compiled operation is done primarily by the setup () function, which requires two parameters: a name parameter indicates which thing to compile, and a list represents the object to compile

setup.py

<pre name= "code" class= "python" >from distutils.core import Setup, Extension

MOD = ' extest '
setup (name= MoD, ext_modules=[
        Extension (mod, sources=[' extest.c ')])

Run setup.py: Execute command python setup.py build

The result is a build directory in the current directory, a extest.so file in the directory, and the module can be import in the script.

method Two: Using cTYPES for C extension

To extend Python, we can write a module with C + +, but that requires enough knowledge of the bottom layer of Python, including the Python object model

Type, commonly used modules, reference count, etc., the threshold is high, and not convenient to use the existing C library. And ctypes the alternative, through the encapsulation

Functions such as Dlopen/dlsym, and provides a wrapper/unpack of the data structures in C, allowing Python to load the dynamic libraries and export the functions straight

be used.

A simple example:

This example uses ctypes directly using the C standard library function without writing C code, and using the C standard library function produces an optimization effect

Script One

Import Timeit
import random

def generate (num): While
    num:
        yield Random.randrange (a)
        num = 1

Print (Timeit.timeit ("sum (Generate (999))", setup= "from __main__ import Generate", number=1000))

<span style= "Font-size:18px;color: #FF0000;" > Script two </span>
import Timeit from
ctypes import Cdll

def generate_c (num):
#Load standard C Library  
       libc = Cdll. LoadLibrary ("libc.so.6") #Linux
       # libc = cdll.msvcrt #Windows while  
       num:
            yield libc.rand ()%
            num -= 1

print (Timeit.timeit ("Sum (Generate_c (999))", setup= "from __main__ import Generate_c", number=1000))

The first script uses Python's random function, which runs about 1.067 seconds. The second script uses the random function of C to run for about 0.423 seconds.

We also use cTYPES to write our own modules to import using:

Step one to create the application code

* * FUNCTIONS.C * * * * * #include "stdio.h" #include "stdlib.h" #include "string.h" * Http://rosettacode.org/wiki/Sorting_algo
    rithms/merge_sort#c */inline void Merge (int *left, int l_len, int *right, int r_len, int *out) {int I, j, K;
        for (i = j = k = 0; I < L_len && J < R_len;) out[k++] = Left[i] < Right[j]?

    Left[i++]: right[j++];
    while (I < l_len) out[k++] = left[i++];
while (J < r_len) out[k++] = right[j++];
    }/* Inner recursion of merge sort */void recur (int *buf, int *tmp, int len) {int l = LEN/2;

    if (Len <= 1) return;
    /* Note that BUF and TMP are swapped/recur (TMP, BUF, L);

    Recur (tmp + L, buf + L, len-l);
Merge (TMP, L, TMP + L, len-l, buf);  }/* Preparation work before recursion/void Merge_sort (int *buf, int len) {/* call alloc, copy and free only once
    * * int *tmp = malloc (sizeof (int) * len);

    memcpy (TMP, buf, sizeof (int) * len);

    Recur (BUF, TMP, Len);
Free (TMP); } INT Fibrec (int n) {if (n < 2) return n;
else return Fibrec (n-1) + Fibrec (n-2);     }
~

Step two: Compile the link to the. so file

Execution instructions: gcc-wall-fpic-c functions.c

Gcc-shared-o libfunctions.so FUNCTIONS.O

Step Three: Use these libraries in your own Python scripts, with the following scripts running in pure Python modules (with Python in front of the output), and the extensions we import (the C before Output) module to run, comparing the execution time of the modules.

functions.py

From ctypes Import *
import time

libfunctions = Cdll. LoadLibrary ("./libfunctions.so")

def fibrec (n):
        if n<2: Return
                n
        else: return
                Fibrec (n-1) + Fibrec (n-2)

start = Time.time ()
fibrec
finish = time.time ()
print ("Python:" + str (finish-start)

#C Fibonacci  
start = Time.time ()
x = Libfunctions.fibrec ()
finish = Time.time ()
print ("C : "+ str (finish-start))

functions2.py

From ctypes Import * Import Time libfunctions = Cdll. LoadLibrary ("./libfunctions.so") #Python Merge Sort from random import shuffle, sample #Generate 9999 random numbers b

Etween 0 and 100000 numbers = sample (range (100000), 9999) Shuffle (numbers) C_numbers = (C_int * len (Numbers)) (*numbers) From HEAPQ import Merge Def merge_sort (m): If Len (m) <= 1:return m middle = len (m)//2 left = M[:middle] right = M[middle:] left = Merge_sort (left) right = Merge_sort (right) return list (left, R  ight)) start = Time.time () numbers = Merge_sort (numbers) finish = Time.time () print ("Python:" + str (finish-start)) #C  Merge Sort start = Time.time () libfunctions.merge_sort (ByRef (C_numbers), Len (numbers) finish = Time.time () print ("C:"                                                                               
+ str (finish-start)) ~ ~                       ~

Run results


You can see that the Pure Python module runs more than 10 times times as long as the extension module we write.

Method III: Using Cython for C expansion (reference Cython three-minute entry)

Cyhton is a tool used to quickly generate the Python extension Module (extention module), which is a hybrid of the Python language syntax and the C language syntax. Exactly speaking, Cython is a separate language, designed to write an extension library for import in Python. In fact, Cython's syntax is basically consistent with Python, and Cython has a special "compiler", first converts the Cython code into C (automatically joins a large stack of C-python APIs), and then compiles the final Python callable module using the C compiler.

One thing to note is that Cython is used to generate C extensions rather than stand-alone programs. All acceleration is done for a function of an existing Python application. The entire application is not rewritten with C or Lisp, and there is no handwriting C extension. Just a simple way to integrate C's speed and C data types into Python functions

Cython code is different from Python and must be compiled. The compilation passes through two stages: (1) Cython compiles the. c file (2) The C compiler compiles the. c file into the. so file, and the. So file indicates the success of the rewrite function, which can be called directly from the Python code.

Python code #p1 .py   import math   def great_circle (LON1,LAT1,LON2,LAT2):        radius = 3956  #miles        x =  math.pi/180.0       a =  (90.0-LAT1) * (x)         b =  (90.0-LAT2) * (x)        theta =  (lon2-lon1) * (x)        c = math.acos ((Math.Cos (a) *math.cos (b))  +                       ( Math.sin (a) *math.sin (b) *math.cos (theta))        return radius*c  

Python code #p1_test. Py Import Timeit Lon1, LAT1, lon2, lat2 = -72.345, 34.323, -61.823, 54.826 num = 500000 T = time It. Timer ("P1.great_circle (%f,%f,%f,%f)"% (LON1,LAT1,LON2,LAT2), "import P1") print "Pure python f Unction ", T.timeit (num)," SEC "Python code # test Results Pure Python function 2.25580382347 secCython: Using the Python math module Python code #c1 .pyx   import math   def great_circle (FLOAT LON1,FLOAT LAT1, FLOAT LON2,FLOAT LAT2):       cdef float radius =  3956.0       cdef float pi = 3.14159265        cdef float x = pi/180.0       cdef  float a,b,theta,c       a =  (90.0-LAT1) * (x)         b =  (90.0-LAT2) * (x)        theta =  ( Lon2-lon1) * (x)        c = math.acos ((Math.Cos (a) *math.cos (b))  +   (Math.sin (a) *math.sin (b) *math.cos (theta))        return radius*c    Python code # setup.py   from distutils.core import setup   from  Distutils.extension import Extension   from cython.distutils import build_ext   Ext_ modules=[       extension ("C1",                  ["C1.pyx"])   ]   setup (      name =  "Demos",     cmdclass = {"Build_ext":  build_ext},      ext_modules = ext_modules  )   

python setup.py build_ext--inplace

Python code #c1_test. Py Import Timeit Lon1, LAT1, lon2, lat2 = -72.345, 34.323, -61.823, 54.826 num = 500000 T = time It. Timer ("C1.great_circle (%f,%f,%f,%f)"% (LON1,LAT1,LON2,LAT2), "Import C1") <