Test with python-High-Performance Testing Tool (3)

Source: Internet
Author: User



Solution 3: Change the system architecture

Before starting a multi-process, I would like to briefly describe python GIL, and I had some misunderstandings about it. Because the mechanism of python GIL exists, there is only one thread running at the same time, but this thread can run on different cores at different times. This scheduling is completed by the operating system, if you write an endless loop and run enough threads, you can clean up the CPU usage of the entire system. In this case, you can see through top in Linux that the CPU usage of us is not large, however, sy occupies a large amount of CPU, which is mainly consumed by system scheduling. The following is the test code. You can try it.

 

import threadingclass MultipleThread(threading.Thread):    def __init__(self):        threading.Thread.__init__(self)    def run(self):        while 1:            print herefor i in xrange(100):    multiple_thread=MultipleThread()    multiple_thread.start()multiple_thread.join()

 

Since GIL can run only one thread at the same time, multithreading can improve efficiency, of course! Opening 3-4 threads can significantly improve the performance. It may increase by about 2-3 times, but increasing the number of threads is a side effect.


Multi-threaded system architecture:

 

There is no bottleneck in sending and receiving. The main bottleneck lies in the red line, decode, and encode sections. Changing multithreading to multi-process is relatively simple and requires little work. You only need to change the information that requires multi-process sharing from Queue to multiprocessing. queue () is enough, and the inherited DiameterMsgParser (threading. thread) to DiameterMsgParser (multiprocessing. process). The log output is troublesome. writing the same file in multiple processes in the logging module of python may cause confusion. This is described separately later.

import multiprocessingimport loggingclass Worker(multiprocessing.Process):    def __init__(self,mp_name,input_queue):        multiprocessing.Process.__init__(self,name=mp_name)        self.input_queue=input_queue    def run(self):        for i in xrange(100):            self.input_queue.put_nowait(i)            logging.debug(test +str(i))

 

After multithreading is changed to multi-process, 4170 meesages (two-way) can be supported on sunfire 2.4 (16 core, 5000G), with a CPU usage of 30-40%, using the standard python2.7, because pypy was not successfully installed on solaris, I did not test the performance impact of pypy on this machine. However, I tested python and pypy on a 2-core linux machine. In the case of multi-process efficiency, pypy did not improve the efficiency by a factor. I did not find any reason, run the following command on a machine with a large number of CPU cores.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.