How to write Python module extensions in C + + (ii)

The Python module contains the class creation (bottom)

1. Method table creation for class

   • Directly on the code

      Static Pymethoddef vcam_methodmembers[] =//The list of all member function structures of the class is also terminated with a full null structure {{"Set_fill", (pycfunction) Vcam_setfill, M  Eth_varargs, "Set Video Resize method (0:aspect fit, 1:aspect Fill, 2:stretch), used when input frame size differs from VCam output size. "}, {" Mirror ", (pycfunction) Vcam_mirror, Meth_varargs," mirror the output video (0:no Mirror, othe Rs:mirror), Non-persistent. "}, {" Rotate ", (pycfunction) vcam_rotate, Meth_varargs," Rotate the input video degree (0:no rotate, others:rotate), Non-persistent. "}, {" Flip ", (pycfunction) Vcam_flip, Meth_varargs," Vertical Flip the Output Video (0:no Flip, Others:flip), Non-persistent. "}, {" Set_difault_image ", (pycfunction) Vcam_setdefaultim Age, Meth_varargs, "Set a 24bits bitmap file as VCam default idle image, which'll be displayed when nothing is being PLA Yed.\ncall it with a NULL parameter or an empty string would reset it to the default one.\n the image would be resized (aspect FIT) only if it ' s bigger than VCam output size. "}, {" Set_name ", (pycfunction) Vcam_setfriendlyname, Meth_varargs," the device's name is \ "Vir Tual camera\ "By default, and you can use it to set a different name."}, {"Set_license", (pycfunction) vcam_setlicense Code, Meth_varargs, "You can set license code here if you ve purchased VCam SDK.  The water mark (TRIAL) is removed with a valid license, and call it with a wrong one would show the watermark again. " }, {"Set_output_format", (pycfunction) Vcam_format, Meth_varargs, "Set display Format (WIDTH,HEIGHT,FPS)."}, { "Send_image", (pycfunction) vcam_sendimg, Meth_varargs, "Display a Image (path) to VCam."}, {"Capture_screen", (Pycfun  ction) Vcam_capturescreen, Meth_varargs, "Capture region of the screen and set it as VCam output."}, {"Get_output_format", (pycfunction)    Vcam_getoutputformat, Meth_noargs, "Get VCam output Video size (640x480 by default), and frame rate (from default)."}, {null, NULL, NULL, NULL}};

   • Definition of PYMETHONDDEF structure

     struct PYMETHODDEF {const char *ml_name;    /* The name of the built-in Function/method */Pycfunction Ml_meth;   /* The C function that <isindex></isindex>mplements it */int ml_flags;    /* combination of METH_XXX flags, which mostly describe the args expected by the C func */    const char *ml_doc; /* The __doc__ attribute, or NULL */};typedef struct pymethoddef pymethoddef; #define PYCFUNCTION_NEW (ML, self) pycfunction                                         _newex (ML), (self), NULL) Pyapi_func (Pyobject *) Pycfunction_newex (PYMETHODDEF *, Pyobject *,  Pyobject *);/* Flag passed to newmethodobject *//* #define Meth_oldargs 0x0000 – Unsupported now */#define Meth_varargs 0x0001#define meth_keywords 0x0002/* Meth_noargs and meth_o must not being combined with the flags above. */#define Meth_noargs 0x0004#define meth_o 0x0008/* Meth_class and meth_static are a little different; These control the construction of MEthods for a class. These cannot is used for functions in modules.  */#define METH_CLASS 0x0010#define meth_static 0x0020/* Meth_coexist allows a method to be entered even though a slot  has already filled the entry. When defined, the flag allows a separate method, ' __contains__ ' for example, to coexist with a defined slots like Sq_co Ntains. */#define Meth_coexist 0x0040

   • There's nothing else to say. Structure definition is clear, that is, the third element ml_falg need to be adjusted according to the timing of the function of the parameters required to be said several common flags other see manuals
     • Meth_noargs indicates that no arguments are passed in,
     • Meth_keywords indicates the incoming keyword parameter
     • Meth_varargs indicates the incoming positional parameter
     • Some flags can be combined to pass in as meth_varargs| Meth_keywords
     • Note Meth_noargs cannot be used in combination with the previous two flags
2. The built-in property information table for the Write class describes the Pytypeobject instance Vcam_classinfo

   • Directly on the code, the code contains most of the elements of the pytypeobject structure, see Object.h header file definition

     Static Pytypeobject vcam_classinfo ={pyvarobject_head_init (NULL, 0) "Pyvcam.vcam",//Can be obtained by __class__ this character   String. cpp can be obtained using the class. __name__.  const char * sizeof (VCAM),//Tp_basicsize class/structure length. Call Pyobject_new When you need to know its size.      py_ssize_t 0,//tp_itemsize py_ssize_t (destructor) vcam_destruct,//Destructor of class.   destructor 0,//class of the print function Printfunc 0,//class GetAttr function Getattrfunc 0,//class SetAttr function Setattrfunc 0,//formerl    Y known as tp_compare (Python 2) or tp_reserved (Python 3) pyasyncmethods * 0,//TP_REPR built-in function calls. Reprfunc 0,//tp_as_number pointer pynumbermethods * 0,//t                              P_as_sequence Pointer pysequencemethods * 0,//tp_as_mapping pointer pymappingmethods * 0, Tp_hash  Hashfunc 0,//tp_call Ternaryfunc 0,//tp_str/print built-in function call. Reprfunc 0,//tp_getattro getattrofunc 0,//tp_setattro set Attrofunc 0,//tp_as_buffer pointer Functions to access object as Input/output buffer Pybufferpro CS Py_tpflags_default | Py_tpflags_basetype,//tp_flags If no method is provided, py_tpflags_defaule unsigned long "VCam Module write  by c++! ",//Tp_doc __doc__, class/structure of the docstring. const char * 0,//tp_traverse call function for all accessible obj ECTS Traverseproc 0,//tp_clear delete references to contain                                                      Ed Objects Inquiry 0,//tp_richcompare Richcmpfunc 0, Tp_weaklistofFset py_ssize_t 0,//tp_iter Getiterfunc 0, Tp_iternext iternextfunc/* Attribute descriptor and subclassing stuff */VCAM_METHODM   Embers,//A collection of all methods of the class.  Pymethoddef * vcam_datamembers,//A collection of all data members of the class.                                              Pymemberdef * 0,//tp_getset pygetsetdef * 0, Tp_base _typeobject * 0,//tp_dict PYOB                                                          ject * 0,//tp_descr_get Descrgetfunc 0, Tp_descr_set Descrsetfunc 0,//tp_dictof Fset py_ssize_t (Initproc) Vcam_init,//constructor for class. Tp_init Initproc 0,//tp_alloc allo   Cfunc 0,//tp_new Newfunc 0,//tp_free Freefunc 0,//TP_IS_GC inquiry}; 

   • Vcam_classinfo the Init functions, destructors, method tables, member tables, etc. created earlier into the Class information table

Module creation and initialization

1. Creating module information

• Directly on the code

     static PyModuleDef ModuleInfo =    {    PyModuleDef_HEAD_INIT,    "PyVcam",               //模块的内置名--__name__.    NULL,                 //模块的DocString.__doc__    -1,    NULL, NULL, NULL, NULL, NULL    };

  • Pymoduledef structural problem definition

    typedef struct PyModuleDef{  PyModuleDef_Base m_base;  const char* m_name;  const char* m_doc;  Py_ssize_t m_size;  PyMethodDef *m_methods;  struct PyModuleDef_Slot* m_slots;  traverseproc m_traverse;  inquiry m_clear;  freefunc m_free;} PyModuleDef;

1. Initializing the module
   • First on code

       pymodinit_func pyinit_pyvcam (void)//module external name is--pyvcam{gdiplus::gdiplusstartupinput Star    Tupinput;    Gdiplusstartup (&m_gdiplustoken, &startupinput, NULL);    pyobject* Preturn = 0;       Vcam_classinfo.tp_new = pytype_genericnew;    The new built-in function of this class-establishes an object.    if (Pytype_ready (&vcam_classinfo) < 0) return NULL;    Preturn = Pymodule_create (&moduleinfo);    if (Preturn = = null) return null;    Py_incref (&vcam_classinfo); Pymodule_addobject (Preturn, "VCam", (pyobject*) &vcam_classinfo);    Add this class to the dictionary of the module. return preturn;}  

   • Code Explanation:
     • The Python module must export a function with a return value of pyobject* named Pyinit_xxx to initialize the module information, and the Python load module will go back and call this function directly to initialize.
     • PYMODINIT_FUNC macro is actually the following statement: __declspec (dllexport) pyobject*
     • Vcam_classinfo.tp_new = pytype_genericnew; This statement can actually not write directly in the front vcam_classinfo in the corresponding position to join the pytype_genericnew can, think to find the corresponding that set to find dizzy simply write it directly in this form; in the front of the whole vcam_classinfo The structure of the back can not be written, directly in the form of vcam_classinfo.xxx =xxx write
     • Call a Pytype_ready (&vcam_classinfo) to complete the definition of the class
     • Then create the module with Pymodule_create (&moduleinfo)
     • Call Pymodule_addobject to add the Vcam class to the module and don't forget to increase the class body reference count
     • Returning the module to Python is done

Pythonc Scaling execution Efficiency issues (GIL)

1.GIL problem
* Gil Lock principle

    for (;;) {        if (--ticker < 0) {   //这是之前版本的GIL锁原理 执行check_interval条数指令 放一次GIL 貌似现在版本不再按照指令条数来放锁了而是按照时间间隔            ticker = check_interval;                         /* Give another thread a chance */            PyThread_release_lock(interpreter_lock);   //释放 GIL                     /* Other threads may run now */                         PyThread_acquire_lock(interpreter_lock, 1); //立马重新申请GIL 一放一抢 其他线程就有机会        }                     bytecode = *next_instr++;  //这里读入python指令        switch (bytecode) {              /* execute the next instruction ... */  //执行指令        }    }* 由于CPython GIL存在在进行多线程任务时 python指令在执行时会一直占着GIL导致其他线程一直在等着抢锁 于是多线程就编程了单线程，无论你开多少个线程貌似都只能同时有一个线程在运行

1. The solution of the Gil lock problem
   In the pure Python environment CPython Gil seems to have no solution, but Gil really no solution?
   • It is known that IO-intensive scenarios can significantly improve execution efficiency by using multithreading, which means that the Gil is freed during IO task execution and obviously this release is definitely not released at Ticker<0, how does the IO task release the Gil?

     • The IO Task release principle is as follows

       /* s.connect((host, port)) method */static PyObject *sock_connect(PySocketSockObject *s, PyObject *addro){    sock_addr_t addrbuf;    int addrlen;    int res;    /* convert (host, port) tuple to C address */    getsockaddrarg(s, addro, SAS2SA(&addrbuf), &addrlen);    Py_BEGIN_ALLOW_THREADS    res = connect(s->sock_fd, addr, addrlen);    Py_END_ALLOW_THREADS    /* error handling and so on .... */}

     • The above is a partial socket code, you can see the call before the implementation of Connect a macro py_begin_allow_threads This macro is used to release the Gil successful connect and then call Py_end_allow_ THREADS re-apply Gil

     • Gil Problem solved

   • Who said that compute-intensive can not be multithreaded, it seems that the use of C + + write a module to deal with computing tasks multithreading can still achieve parallel effects
     • Let's start writing code to verify the problem

     • There are two compute-intensive functions written in the C + + module, and the only difference is that the function calculates the return of the algorithm without distinction: a function in the mid-term is freed before the Gil computation is completed. Re-request lock

        static PyObject* Gil_free(GilTest* self,PyObject* args){        LONGLONG num;    if (!PyArg_ParseTuple(args, "L", &num))return NULL;    LONGLONG rst;    Py_BEGIN_ALLOW_THREADS        for (LONGLONG i = 1; i <= num * 100; i++)        {            for (LONGLONG j = 1; j <= num * 100; j++)            {                rst = i*j;            }        }                    Py_END_ALLOW_THREADS        return Py_BuildValue("i", rst);}static PyObject* Gil_lock(GilTest* self, PyObject* args){    LONGLONG num;    if (!PyArg_ParseTuple(args, "L", &num))return NULL;     LONGLONG rst;    for (LONGLONG i = 1; i <= num * 100; i++)        {            for (LONGLONG j = 1; j <= num * 100; j++)            {                 rst = i*j;            }        }    return Py_BuildValue("i", rst);}

     • Encapsulates a function into a Python module calling the module write a script open multithreaded execution

         from giltest import giltestimport timefrom threading import threaddef foo (num, I, start): obj = Giltest ()    Obj.compute_with_gil (NUM) # Called when the function is evaluated without releasing Gil print ("foo%s is over"% i, time.time ()-start) def bar (num, I, start): obj = Giltest () Obj.compute_without_gil (num) # The function that is called is computed when the Gil print is released ("bar%s is over"% i, time.time ()-Start) de F run (): Print ("stat foo") start = Time.time () # Open foo thread starts timing thread_list1 = [] for i in range: thre Ad_list1.append (Thread (Target=foo, args= (+, I, start)))) for I in Thread_list1:i.start () for I in thread_l    Ist1:i.join () print ("Stat bar") time.sleep (1) start = Time.time () # Open bar thread start timing thread_list2 = []        For I in range: Thread_list2.append (thread (Target=bar, args= (+, I, start))) for I in Thread_list2: I.start () for I in Thread_list2:i.join () if __name__ = = ' __main__ ': Run ()  

     • Output execution Results

       stat foofoo 0 is over 2.2932560443878174foo 1 is over 4.577575445175171foo 2 is over 6.859208583831787foo 3 is over 9.145148277282715foo 4 is over 11.43115520477295foo 5 is over 13.71883225440979foo 6 is over 15.999829292297363foo 7 is over 18.281397581100464foo 8 is over 20.57776975631714foo 9 is over 22.851707935333252stat barbar 3 is over 4.594241380691528bar 6 is over 4.594241380691528bar 7 is over 4.609868288040161bar 2 is over 4.63910174369812bar 8 is over 5.750362157821655bar 4 is over 5.765988826751709bar 5 is over 5.859748840332031bar 0 is over 5.859748840332031bar 1 is over 5.859748840332031bar 9 is over 5.937881946563721Process finished with exit code 0

     • It can be found that the Foo thread is exactly like running a single thread, and each of the threads takes about 2.3 seconds more to complete than the previous thread, while the bar thread is really multi-threaded, the thread finishes the calculation time difference is very small, and the completion sequence is disorderly, Because the CPU is quad-core, there is still a CPU grab between threads, and each thread runs a bit longer than Foo (foo each thread's operation is almost exclusively running)

     • And take a look at CPU usage.
       

     When executing foo, the Python process consumes about 15% of the CPU, and when the program executes to the bar thread, it can see that the Python process's CPU usage is up to nearly 100% in the line, which also shows that the Python thread is parallel at this point.

How to write Python module extensions in C + + (ii)