Test with python-High-Performance Testing Tool (3)

Solution 3: Change the system architecture

Before starting a multi-process, I would like to briefly describe python GIL, and I had some misunderstandings about it. Because the mechanism of python GIL exists, there is only one thread running at the same time, but this thread can run on different cores at different times. This scheduling is completed by the operating system, if you write an endless loop and run enough threads, you can clean up the CPU usage of the entire system. In this case, you can see through top in Linux that the CPU usage of us is not large, however, sy occupies a large amount of CPU, which is mainly consumed by system scheduling. The following is the test code. You can try it.

 

import threadingclass MultipleThread(threading.Thread):    def __init__(self):        threading.Thread.__init__(self)    def run(self):        while 1:            print herefor i in xrange(100):    multiple_thread=MultipleThread()    multiple_thread.start()multiple_thread.join()

 

Since GIL can run only one thread at the same time, multithreading can improve efficiency, of course! Opening 3-4 threads can significantly improve the performance. It may increase by about 2-3 times, but increasing the number of threads is a side effect.

Multi-threaded system architecture:

 

There is no bottleneck in sending and receiving. The main bottleneck lies in the red line, decode, and encode sections. Changing multithreading to multi-process is relatively simple and requires little work. You only need to change the information that requires multi-process sharing from Queue to multiprocessing. queue () is enough, and the inherited DiameterMsgParser (threading. thread) to DiameterMsgParser (multiprocessing. process). The log output is troublesome. writing the same file in multiple processes in the logging module of python may cause confusion. This is described separately later.

import multiprocessingimport loggingclass Worker(multiprocessing.Process):    def __init__(self,mp_name,input_queue):        multiprocessing.Process.__init__(self,name=mp_name)        self.input_queue=input_queue    def run(self):        for i in xrange(100):            self.input_queue.put_nowait(i)            logging.debug(test +str(i))

 

After multithreading is changed to multi-process, 4170 meesages (two-way) can be supported on sunfire 2.4 (16 core, 5000G), with a CPU usage of 30-40%, using the standard python2.7, because pypy was not successfully installed on solaris, I did not test the performance impact of pypy on this machine. However, I tested python and pypy on a 2-core linux machine. In the case of multi-process efficiency, pypy did not improve the efficiency by a factor. I did not find any reason, run the following command on a machine with a large number of CPU cores.