Multithreading C calling the Python API trap

http://blog.csdn.net/networm3/article/details/42423393

It is well known that a service (Wsgi interface) written in scripting languages requires a server container, common such as PHP php-fpm, LIGHTD, and so on. The uwsgi,uwsgi used in Python is a new protocol based on WSGI that can be used to deploy script programs such as Python. However, there may be some unexpected problems with developing a code architecture that is unfamiliar to UWSGI and the API of C calling Python.

Let's look at a piece of code, the following code is the flask frame, each request will be the value of the count is first minus again and again, and then multiply by two. If the request is 50 times, the final result should be 2 power of 50 times.

[python] view plain copy from flask import flask, request      count = 1       app = flask (__name__)        @app. Route ('/test_uwsgi ')    def index ():       global  count       COUNT=COUNT-1       COUNT=COUNT+1        COUNT=COUNT*2       print COUNT       return  ' OK '   

[plain] view plain copy 17179869184 34359738368 68719476736 137438953472 274877906944 549755813888 10 99511627776 2199023255552 4398046511104 8796093022208 17592186044416 35184372088832 70368744177664 140737488 355328 281474976710656 562949953421312 1125899906842624

This is the result of the last few rows that are executed directly by the 50-second index function, and the result is 50 powers of 2. [plain] view plain copy 536870912 1073741824 2147483648 4294967296 8589934592 17179869184 343597383 68 68719476736 137438953472 274877906944 549755813888 1099511627776 2199023255552 4398046511104 8796093 022208 17592186044416 35184372088832 70368744177664 140737488355328 281474976710656 562949953421312112589990 6842624

This is the result of the last few lines obtained with multiple concurrent access to the/TEST_UWSGI interface through AB testing. It can be seen that the final result must be an anomaly number. Why does the program run in Uwsgi with an exception?

In fact, by reading this simple example, you can find that this example is commonly used to illustrate the problem of multithreading shared data synchronization, if not locked will expose the problem example. For the following code, we'll add a mutex when we modify the shared resource count to see if there are any changes. [python] view plain copy from flask import flask, request   import  threading      mutex = threading. Lock ()    count = 1       app = flask (__name__)        @app. Route ('/test_uwsgi ')    def index ():       global  COUNT       global mutex        Mutex.acquire ()        COUNT=COUNT-1       count= count+1       COUNT=COUNT*2       print count        mutex.release ()        return  ' OK '    

The above code is also put into the Uwsgi container running, through the HTTP interface multiple concurrent access 50 times, the result is correct. But that's why. In our original Python code, we didn't write any operations involving multiple processes, although uwsgi in the configuration file enabled multiple threads to handle requests concurrently, but as I originally understood, it was not up to each thread to execute its own separate Python interpreter. The data for each thread when it runs the Python script should not be isolated.

To understand the above problems, we have to study the structure and design of UWSGI and its server architecture.

UWSGI is a server application container that is widely used in python, similar to the server application container for WSGI protocols that are common in PHP, such as mod-php, PHP-FPM, LIGHTD, and so on. The UWSGI agreement is to add a set of UWSGI agreements on top of the original WSGI agreement.

By studying Uwsgi's source code (CORE/UWSGI.C core/loop.c core/init.c core/master_util.c core/ UTIL.C), you can know Uwsgi's server design, the introduction of the Unx book introduces the induction of the server programming Paradigm 8, and TCP pre-create Thread server program, each thread accept.[Python] View Plain copy Int main (int argc, char *argv[], char *envp[])  {       uwsgi_setup (ARGC,&NBSP;ARGV,&NBSP;ENVP);       return  uwsgi_run ();  }      Void uwsgi_setup (int argc, char * Argv[], char *envp[])  {          int i;          struct utsname uuts;           ......         &nbsp .... Set up and initialize various resources, here is omitted, interested to look at yourself               ......           //The main thing is this line of        uwsgi_start (void  *)  uwsgi.argv;  }      int uwsgi_start (VOID&NBSP;*V_ARGV)  {           ......           Simplified Summary Some of the main code            ......          ...  digression, here is to create a multithreaded shared memory space, behind Uwsgi_setup_ Workers will be used when the time comes.             because Uwsgi has a master process that can monitor the status of each child process, it requires an anonymous shared memory         // initialize sharedareas       uwsgi_ Sharedareas_init ();          // setup queue        if  (uwsgi.queue_size > 0)  {            uwsgi_init_queue ();       }           ...  It's important here, UWSGI.P is an interface, where apps deployed in Uwsgi are initialized (in Uwsgi, deployed apps require plug-ins for the corresponding language, such as Python plug-ins)              will also see that, in fact, the Python code that UWSGI executes, the import of all of its modules is executed here &NBsp;      // initialize request plugin only if workers  or master are available       if  (uwsgi.sockets | |  uwsgi.master_process | |  uwsgi.no_server | |  uwsgi.command_mode | |  uwsgi.loop)  {           for  (i = 0;  i < 256; i++)  {                if  (uwsgi.p[i]->init)  {                    uwsgi.p[i]->init ();                }            }       }          // again  check for workers/sockets...       if  (uwsgi.sockets | |  uwsgi.master_process | |  uwsgi.no_server | |  uwsgi.command_mode | |  uwsgi.loop)  {           for  (i = 0;  i < 256; i++)  {                if  (uwsgi.p[i]->post_init)  {                    uwsgi.p[i]->post_init ();                }            }       }           ... This is primarily to set up shared memory space for each worker        // initialize workers/master shared  memory segments   &NBSP;&NBSP;&NBSP;&NBsp;uwsgi_setup_workers ();          // here we spawn  the workers...       if  (!uwsgi.status.is_cheap)  {            if  (uwsgi.cheaper &&  Uwsgi.cheaper_count)  {                int nproc = uwsgi.cheaper_initial;                if  (!nproc)                     nproc = uwsgi.cheaper_count;                for  (i = 1; i <=  uwsgi.numproc; i++)  {                    if&nbsp. (i <= nproc)  {                        if  (Uwsgi_respawn_worker (i))                             break;                        uwsgi.respawn_delta = uwsgi_now ();                    }                     else {                        uwsgi.workers[i].cheaped = 1;                    }                }           }            else {                for  (i = 2 - uwsgi.master_process; i <  uwsgi.numproc + 1; i++)  {                     ... Here is the number of processes that we set up to fork the sub process                     if  (Uwsgi_respawn_worker (i))                         break;                    uwsgi.respawn_delta = uwsgi_now ();                }           }        }             // END OF  initialization       return 0;     }       Int uwsgi_respawn_worker (int wid)  {           ... Mainly this line of code, fork process, the inside will not follow the        pid_t pid = uwsgi_fork (uwsgi.workers[ Wid].name);          if  (pid == 0)  {           signal (sigwinch, worker_wakeup);            signal (sigtstp, worker_wakeup);            uwsgi.mywid = wid;            Uwsgi.mypid = getpid ();           // pid  is updated by the master           // uwsgi.workers[uwsgi.mywid].pid = uwsgi.mypid;            // OVERENGINEERING  (just to be safe)             uwsgi.workers[uwsgi.mywid].id = uwsgi.mywid;            /*               uwsgi.workers[uwsgi.mywid].harakiri = 0;               uwsgi.workers[uwsgi.mywid].user_harakiri = 0;               uwsgi.workers[uwsgi.mywid].rss_size = 0;               uwsgi.workers[uwsgi.mywid].vsz_size = 0;             */           //  do not reset worker counters on reload !!!            //uwsgi.workers[uwsgi.mywid].requests =  0;           // ...but maintain a delta  counter  (yes this is racy in multithread)             //uwsgi.workers[uwsgi.mywid].delta_requests = 0;            //uwsgi.workers[uwsgi.mywid].failed_requests = 0;    &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBsp; //uwsgi.workers[uwsgi.mywid].respawn_count++;            //uwsgi.workers[uwsgi.mywid].last_spawn = uwsgi.current_time;           }       else if  (pid < 1)  {            uwsgi_error ("fork ()");        }       else {            // the pid is set only in the master, as the  worker should never use it            uwsgi.workers[wid].pid = pid;               if  (respawns > 0)  {                uwSgi_log ("respawned uwsgi worker %d  (new pid: %d) \ n", wid,  (int)   PID);           }            else {                uwsgi_log ("spawned uwsgi worker %d  (pid: %d, cores: %d) \",  wid,  pid, uwsgi.cores);      &nbs