Nested python interpreter (Embedding Python in Another Application)

Hello, world

[dongsong @ bogon python_study] $ cat py.cpp
#include <Python.h>
int main (int argc, char ** argv)
{
        Py_Initialize ();
        PyRun_SimpleString ("import sys");
        PyRun_SimpleString ("sys.path.append ('./')");
        PyObject * pModule = PyImport_ImportModule ("helloword");
        PyObject * pFunc = PyObject_GetAttrString (pModule, "hello");
        PyEval_CallObject (pFunc, NULL);
        Py_Finalize ();
        return 0;
}
[dongsong @ bogon python_study] $ cat helloword.py
def hello ():
        print 'hello, world!'
[dongsong @ bogon python_study] $ g ++ -o py py.cpp -I / home / dongsong / venv / include / python2.6 / -lpython2.6
[dongsong @ bogon python_study] $ ./py
hello, world!
Official original document
http://docs.python.org/3/extending/embedding.html
5. Embedding Python in Another Application
In the previous chapter we discussed how to extend Python, that is, how to extend Python's functionality by connecting to a C library. There is another way to do this: by embedding Python to enrich our C / C ++ applications. Embedding allows our program to use Python to achieve certain functions, rather than using only C or C ++. This has multiple uses; an example is to allow users to customize the application we need by writing some Python scripts. If some functions are easier to write in Python, we can also consider using it.


Embedded Python and extended Python are similar, but not identical. The difference is that when we extend Python, the main program is still a Python interpreter, and if we embed Python, the main program has nothing to do with Python-on the contrary, only some parts of the application occasionally call the Python interpreter to run Python Code.


So if we want to embed Python, we need to provide our own main program. One of the things the main program needs to do is initialize the Python interpreter. At the very least, we have to call the function Py_Initialize (). We can also choose to pass the command line parameters to Python. Then we can call the interpreter from anywhere in the application.


There are several different ways to call the interpreter: pass the string containing the Python statement to PyRun_SimpleString (); or pass a stdio file pointer and file name (just for error message identification) to PyRun_SimpleFile (). We can also Call the lower-level operations described in the previous section to construct and use Python objects.


5.1. Very High Level Embedding

The simplest form of embedded Python is to use the high level interface (the verf high level interface). This interface executes Python scripts without direct interaction with the application. This applies to the case of operating on files.

#include <Python.h>

int
main (int argc, char * argv [])
{
  Py_SetProgramName (argv [0]); / * optional but recommended * /
  Py_Initialize ();
  PyRun_SimpleString ("from time import time, ctime \ n"
                     "print ('Today is', ctime (time ())) \ n");
  Py_Finalize ();
  return 0;
}
Py_SetProgramName () should be called before Py_Initialize () to tell the interpreter the path to the Python run-time libraries. Next, the Python interpreter is initialized with Py_Initialize (), and then executes the hard-coded Python script and outputs the date and time. Next, Py_Finalize () closes the interpreter and the program exits. For an actual program, we may need to obtain the python code from another place, which may be a text editor, a file, or a database. It is more appropriate to use PyRun_SimpleFile () to get Python code from a file. It saves us the trouble of allocating memory and reading the contents of the file.

5.2. Beyond Very High Level Embedding: An overview


The high-level interface allows us to execute arbitrary code fragments in the application, but exchanging data is quite troublesome. If required, we should use lower level calls. Although some more C code has been written, we can do more things.


Note that, except for different purposes, extended Python and embedded Python are quite similar (is quite the same activity). Most of the topics discussed in previous chapters are still valid here. To demonstrate this, consider what the C code for extending Python actually does:


1. Convert data from Python to C,

2. Perform a function call to a C routine using the converted values with the converted data,

3. Convert the function return value from C to Python



Embedded Python, interface code to do:

1. Convert data from C to Python,

2. Use the converted data to perform a function call to a Python interface routine using the converted values,

3. Convert the function return value from Python to C

As you can see, this step of data conversion is just a simple wrapper for cross-language delivery in different directions. The only difference is the program that is called between the two data conversions. We call a C program when expanding, and we call a Python program when it is embedded.


In this chapter we will not discuss how to convert data from Python to C or vice versa. The reasonable use of quotes and error handling also assume that we all understand. Because these are no different from the extended interpreter, we can refer to the previous chapters to understand this information.


5.3. Pure Embedding

The goal of the first program is to execute a function in a Python script. As explained in the high-level interface section, the Python interpreter does not directly interact with the application (the content explained in the next step will interact).


The following code executes the function in the Python script:

#include <Python.h>

int
main (int argc, char * argv [])
{
    PyObject * pName, * pModule, * pDict, * pFunc;
    PyObject * pArgs, * pValue;
    int i;

    if (argc <3) {
        fprintf (stderr, "Usage: call pythonfile funcname [args] \ n");
        return 1;
    }

    Py_Initialize ();
    pName = PyUnicode_FromString (argv [1]);
    / * Error checking of pName left out * /

    pModule = PyImport_Import (pName);
    Py_DECREF (pName);

    if (pModule! = NULL) {
        pFunc = PyObject_GetAttrString (pModule, argv [2]);
        / * pFunc is a new reference * /

        if (pFunc && PyCallable_Check (pFunc)) {
            pArgs = PyTuple_New (argc-3);
            for (i = 0; i <argc-3; ++ i) {
                pValue = PyLong_FromLong (atoi (argv [i + 3]));
                if (! pValue) {
                    Py_DECREF (pArgs);
                    Py_DECREF (pModule);
                    fprintf (stderr, "Cannot convert argument \ n");
                    return 1;
                }
                / * pValue reference stolen here: * /
                PyTuple_SetItem (pArgs, i, pValue);
            }
            pValue = PyObject_CallObject (pFunc, pArgs);
            Py_DECREF (pArgs);
            if (pValue! = NULL) {
                printf ("Result of call:% ld \ n", PyLong_AsLong (pValue));
                Py_DECREF (pValue);
            }
            else {
                Py_DECREF (pFunc);
                Py_DECREF (pModule);
                PyErr_Print ();
                fprintf (stderr, "Call failed \ n");
                return 1;
            }
        }
        else {
            if (PyErr_Occurred ())
                PyErr_Print ();
            fprintf (stderr, "Cannot find function \"% s \ "\ n", argv [2]);
        }
        Py_XDECREF (pFunc);
        Py_DECREF (pModule);
    }
    else {
        PyErr_Print ();
        fprintf (stderr, "Failed to load \"% s \ "\ n", argv [1]);
        return 1;
    }
    Py_Finalize ();
    return 0;
}
The above code loads the Python script specified by argv [1] and calls the function named by argv [2]. The integer parameters of the function are other values of the argv array. If we compile and link (compile and link, http: // docs.python.org / 3 / extending / embedding.html # compiling) This program (the final executable file is named "call"), use it to execute a Python script, such as:

def multiply (a, b):
    print ("Will compute", a, "times", b)
    c = 0
    for i in range (0, a):
        c = c + b
    return c
Then the result should be:

$ call multiply multiply 3 2
Will compute 3 times 2
Result of call: 6
Although the program is large enough for its function, most of the code is doing data conversion between Python and C, and error reporting. What is interesting is that it complies with the rules of embedded Python, starting with the following code (initializing the interpreter):

Py_Initialize ();
pName = PyUnicode_FromString (argv [1]);
/ * Error checking of pName left out * /
pModule = PyImport_Import (pName);
After initializing the interpreter, the script is imported with PyImport_Import () (http://docs.python.org/3/c-api/import.html#PyImport_Import). This function requires a Python string as its parameter. The Python string is constructed using the PyUnicode_FromString () (http://docs.python.org/3/c-api/unicode.html#PyUnicode_FromString) data conversion function .

pFunc = PyObject_GetAttrString (pModule, argv [2]);
/ * pFunc is a new reference * /

if (pFunc && PyCallable_Check (pFunc)) {
    ...
}
Py_XDECREF (pFunc);
Once the script is loaded, the name you need to find can be obtained through PyObject_GetAttrString () (http://docs.python.org/3/c-api/object.html#PyObject_GetAttrString). If the name exists and the returned object is callable, we can safely assume that it is a function. Then the program executes normally, constructing a tuple parameter. Then, call the Python function:

pValue = PyObject_CallObject (pFunc, pArgs);
After the function returns, whether pValue is NULL or contains a reference to the value returned by the function. Remember to release the reference after checking the return value!

5.4. Extending Embedded Python

So far, the nested Python interpreter has not accessed the application's own functions. The Python API provides this access by extending the Python interpreter. In other words, the embedded Python interpreter has been extended with functions provided by the application. It sounds complicated, but it's not so exaggerated. Forget about the application to start the Python interpreter. Instead, treat the application as a set of subroutines and write some glue code to allow Python to access these programs, just like writing a normal Python extension. E.g:

static int numargs = 0;

/ * Return the number of arguments of the application command line * /
static PyObject *
emb_numargs (PyObject * self, PyObject * args)
{
    if (! PyArg_ParseTuple (args, ": numargs"))
        return NULL;
    return PyLong_FromLong (numargs);
}

static PyMethodDef EmbMethods [] = {
    {"numargs", emb_numargs, METH_VARARGS,
     "Return the number of arguments received by the process."},
    {NULL, NULL, 0, NULL}
};

static PyModuleDef EmbModule = {
    PyModuleDef_HEAD_INIT, "emb", NULL, -1, EmbMethods,
    NULL, NULL, NULL, NULL
};

static PyObject *
PyInit_emb (void)
{
    return PyModule_Create (& EmbModule);
}
Insert the above code before the main () function. At the same time, insert the following two lines of code before the call to Py_Initialize ().

numargs = argc;
PyImport_AppendInittab ("emb", & PyInit_emb);
These two lines initialize the numargs variable and make the emb.numargs () function accessible to the embedded Python interpreter. With these extensions, Python scripts can do the following things:

import emb
print ("Number of arguments", emb.numargs ())
In an actual application, these methods will expose an API of the application to Python.

5.5. Embedding Python in C ++

It is also possible to embed Python into a C ++ program; as for how to implement it, it depends on the details of the system C ++; generally speaking, we need to write the main program in C ++ and compile and link our program with a C ++ compiler. No need to recompile Python with C ++.


5.6. Compiling and Linking under Unix-like systems

For embedding the Python interpreter in our application, there is no need to specifically find the right flags to pass to your compiler (and linker) because the library modules that Python needs to load C dynamic extensions.


To find out the required compiler and linker tags, we can execute the PythonX.Y-config script, which is generated as part of the installation process (python3-config script may also be available). This script has several options, among which the following items will be more direct for us:


pythonX.Y-config --cflags provides recommended flags for our compilation:

$ /opt/bin/python3.3-config --cflags
-I / opt / include / python3.3m -I / opt / include / python3.3m -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes
pythonX.Y-config --ldflags provides recommended tags for our links:

$ /opt/bin/python3.3-config --ldflags
-L / opt / lib / python3.3 / config-3.3m -lpthread -ldl -lutil -lm -lpython3.3m -Xlinker -export-dynamic
Note: In order to avoid confusion between multiple installed Python (especially system Python and our own compiled Python), in the above example, we should use the absolute path when using pythonX.Y-config.


If this program is not easy to use (this program is not guaranteed to work on all Unix-like platforms; however, we welcome reports of bugs (http://docs.python.org/3/bugs.html#reporting-bugs )), We will have to read our system's documentation on dynamic links and / or check Python's Makefile (using sysconfig.get_makefile_filename () (http://docs.python.org/3/library/sysconfig.html# sysconfig.get_makefile_filename) to find its location) and compile options. In this case, the sysconfig (http://docs.python.org/3/library/sysconfig.html#module-sysconfig) module is a useful tool, this tool can help us extract the programmatically what we want The value of the configuration merged together. E.g:

>>> import sysconfig
>>> sysconfig.get_config_var ('LIBS')
'-lpthread -ldl -lutil'
>>> sysconfig.get_config_var ('LINKFORSHARED')
'-Xlinker -export-dynamic'