Python Multi-core utilization test

For many enterprises, with the passage of time, user volume or enterprise Building point expansion, the use of ERP will appear more and more slow application access situation, in fact, this situation is not limited to ERP, as long as there is data volume growth of the internet business will inevitably encounter, because the beginning is not doing a good job of big data volume of access.

Odoo ERP is developed by Python, Python relative C, C + +, Java and so on in performance is really a lot lower, in the final analysis is originally Python is C with the language developed by the three C + +, in addition to Python people criticized the Global interpreter Lock (Gil, Global interpreter lock), you can access http://cenalulu.github.io/python/if you want to know more about Gil Gil-in-python/?utm_source=tuicool&utm_medium=referral

There was a period of time in the company, OpenERP has developed, and let individual enterprises in use, of course, this process has some enterprises have been reflected in the slow, this period also done a lot of testing, found that indeed openerp in the running process can not play a multi-core situation, basically CPU a core 100% The situation is stuck, but there are at least a good number of core resources are not used AH? Is it just a waste of it? Also try to forward with Nginx, or find a core 100% case.

After oneself also tried to write a multi-threaded network communication program, with the loop constantly pull to the service side of the thread, and then to observe the CPU situation, the surprise found that the CPU really can use more cores, ah, it is said to use Java to develop new business procedures?

Although Java is actually faster than Python in performance, it is still a sense of understanding to start Python's multi-process, multi-threaded test:

Python multi-process (tested with a dead loop):

#!/bin/python

From multiprocessing import Process

Def mulprocess ():

While 1:

Pass

if __name__ = = ' __main__ ':

T1 = Process (target=mulprocess)

t2 = Process (target=mulprocess)

T3 = Process (target=mulprocess)

T4 = Process (target=mulprocess)

T1.start ()

T2.start ()

T3.start ()

T4.start ()

T1.join ()

T2.join ()

T3.join ()

T4.join ()

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650 "this.width=650;" src= "http ://s4.51cto.com/wyfs02/m01/7f/18/wkiol1ctlqfa-2fbaabfsgjyblc678.png "title=" Clipboard1.png "style=" Float:none; " alt= "Wkiol1ctlqfa-2fbaabfsgjyblc678.png"/>

Code set up 4 processes, and then always loop, see the 4 CPU are used to, in fact, white Python is actually can be used to multi-core

Python Multithreading:

#!/bin/python

From threading Import Thread

Def threadprocess ():

While True:

Pass

if __name__ = = ' __main__ ':

T1 = Thread (target=threadprocess)

T1.start ()

T2 = Thread (target=threadprocess)

T2.start ()

T3 = Thread (target=threadprocess)

T3.start ()

T4 = Thread (target=threadprocess)

T4.start ()

T5 = Thread (target=threadprocess)

T5.start ()

T6 = Thread (target=threadprocess)

T6.start ()

T1.join ()

T2.join ()

T3.join ()

T4.join ()

T5.join ()

T6.join ()

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650 "this.width=650;" src= "http ://s1.51cto.com/wyfs02/m00/7f/1b/wkiom1ctleucg67oaabomfl6ee4403.png "title=" Clipboard2.png "style=" Float:none; " alt= "Wkiom1ctleucg67oaabomfl6ee4403.png"/>

Multi-threaded code to build 6 threads, in fact, more than 4 core, but so no problem, a total of 4 cores more than the same run, the CPU display basically used 4 core, but it is recommended not to use multi-threaded, why? 1. Multithreading different multi-process, multi-threaded data in different threads can be shared, CPU resource sharing can not want you to think, in the multi-threaded run is that idle run which, very prone to data chaos situation (if there are changes in the case of data), 2. Of course, multithreading can ensure that the data is unified, Just to add a threading. Lock (), that is to add a lock, want to run the time must first obtain lock lock, this is not the essence of multi-threaded fast processing performance reduced it? What a real egg. On the process, not much, if not the abnormal business process needs to the process of data access, basically the process is run independently, but the system in the establishment of a process and establish a thread of the resource consumption is different, obviously the process is much larger than the thread. Sure enough, this world of what fish and bear can not have the same thing or less AH.

Well, to this end, basically a Python multi-threaded, multi-process situation has a relatively objective understanding. Here's a test of OPENERP's multi-core utilization.

In fact, the new version of the ODOO8, 9 are supported Gevent, multiprocess, threading operation, testing is just gevent with multiprocess, as for threading do not go to test, Unless you are using a single-core CPU. The author uses the OpenERP 7 and Odoo 8 to do the test, here is the main said Odoo 8,odoo 8 environment is CENTOS7, directly with Odoo rep file, yum install, I am more lazy, of course, can use the source of the way, this is not detailed introduction, Do not understand crossing net go. The benefit of Yum installation is that it's basically missing any dependencies you've installed.

First, test the multiprocess before the test directly modify the configuration file/etc/odoo/openerp-server.conf, set workers = 4 (I am the 4 core virtual machine), here to say the setting of the workers situation, If you set workers = 0, then there is only one odoo process, if workers =1 immediately become 4 processes, there is a openerp-gevent process, the most if the use of Odoo chat customers, there is installed gevent under the circumstances will appear , of course, can be removed in the openerp-server.conf configuration file, so to get rid of this process is theoretically workers =1 pull up 3 processes, and then workers =2 is 5 processes and so on, starting systemctl start Odoo.service, first in the most primitive way, open the URL to press and hold F5 constantly refreshed, while observing, back also used ./webbench-c 1000-t http://192.168.1.125/to test

:650) this.width=650; "Src=" Http://s4.51cto.com/wyfs02/M02/7F/1B/wKiom1cTle2wAGgLAABZs9-7TnU162.png " Style= "Float:none;" title= "Clipboard3.png" alt= "Wkiom1ctle2wagglaabzs9-7tnu162.png"/>

The 4 core is used, and the multi-core utilization of Python's multi-process is verified.

Second, next Test gevent,gevent is python an asynchronous concurrency framework, with tornado a little acquaintance, but with the concept of the association, if there is time to find gevent source to see, but I here mainly imitate multi-process, idea: 4 core virtual machine, Open 4 openerp-gevent process, through different ports to access, but I have unified 80 port access, the use of nginx as front-end forwarding, but also test the Nginx multi-core usage, of course, Nginx itself is also asynchronous, After the test has to admit to add Nginx in the visit is really better than not to add a lot, here affixed to the nginx part of the configuration:

XXXXXXXXXXXXXXX # Many did not post it

Worker_processes 4;

Worker_cpu_affinity 0001 0010 0100 1000;

Events {

Use Epoll;

Worker_connections 65563;

}

XXXXXXXXXXXXXXX # Many did not post it

server {

Listen default_server;

Listen [::]:80 default_server;

server_name _;

Root/usr/share/nginx/html

Location/{

Proxy_pass Http://openerp;

}

Xxxxxxxxxxxxxxx

Upstream openerp{

Server 192.168.1.125:8072;

Server 192.168.1.125:8073;

Server 192.168.1.125:8074;

Server 192.168.1.125:8075;

}

Xxxxxxxxxxxxxxxxx

Start four /usr/bin/openerp-gevent-c openerp-server.conf

/usr/bin/openerp-gevent-c openerp-server8073.conf

/usr/bin/openerp-gevent-c openerp-server8074.conf

/usr/bin/openerp-gevent-c openerp-server8075.conf

Start Nginx:systemctl start Nginx.serivce

View Port: 650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/ Zh-cn/images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650) this.width=650; "src = "Http://s4.51cto.com/wyfs02/M02/7F/18/wKioL1cTlqrwxKRVAAAyHcInDo0098.png" title= "clipboard4.png" style= "float: none; "alt=" Wkiol1ctlqrwxkrvaaayhcindo0098.png "/>

OK, test:./webbench-c 1000-t http://192.168.1.125/

1000 concurrent, lasts 30 seconds

650) this.width=650; "src="/e/u261/themes/default/images/spacer.gif "style=" Background:url ("/e/u261/lang/zh-cn/ Images/localimage.png ") no-repeat center;border:1px solid #ddd;" alt= "Spacer.gif"/>650 "this.width=650;" src= "http ://s2.51cto.com/wyfs02/m00/7f/1b/wkiom1ctle_g9n0naab4gcdvpcy471.png "title=" Clipboard5.png "style=" Float:none; " alt= "Wkiom1ctle_g9n0naab4gcdvpcy471.png"/>

Conclusion: Python can use multi-process scenarios to achieve multi-core utilization, Java in performance is better than Python, but Python development speed is relatively fast, odoo can actually achieve multi-core utilization, estimated performance will be a step, In addition nginx multi-core utilization can only support to 8 core (online said, lazy to test).

This article is from the "Fengyunsen" blog, make sure to keep this source http://samfeng.blog.51cto.com/52272/1764852

Python Multi-core utilization test