Performance analysis and optimization case based on Web application

First, dynamic content-based site optimization case

1. Website Operating Environment Description
Hardware environment: 1 IBM X3850 servers, single dual-core Xeon 3.0G CPU,2GB memory, 3 72GB SCSI disks.
Operating system: CentOS5.4.
Site Architecture: The Web application is based on the lamp architecture, and all services are deployed on a single server.
2. Performance problem phenomenon and treatment measures
description of the phenomenon
Site at around 10 o'clock in the morning and 3 o'clock in the afternoon Access peak, Web page can not be opened, restart the service, the site will be able to serve in a period of time, but after a while it becomes slow response, the final page can not be opened completely.
Check Configuration
First check the system resource status, discover the service failure when the system load is very high, the memory is basically exhausted, and then check the Apache configuration file httpd.conf, found that the "maxclients" option value is set to 2000, and open the Apache keepalive feature.
Handling Measures
Based on the above check, the initial judgment is that Apache's "maxclients" option is improperly configured because the system memory is only 2GB in size and the "maxclients" option is configured to 2000, and too many user access processes run out of system memory; Modify the "maxclients" option of the httpd.conf configuration file to reduce this value from 2000 to 1500; Continue to observe that the site is still frequent downtime, and then the "maxclients" option value to 1024, observe a period of time to find, Website service outage time interval extended, not as often as before, but the system load is very high, the page access speed is very slow.
3. First-time analysis optimization
Since it is caused by the exhaustion of the system resources, the website service loses its response, then the use of system resources is analyzed in depth, through the joint use of uptime, Vmstat, top, PS and other commands, the following conclusions are drawn:
Conclusion Description
System average load is high, through the uptime output of the system "load average" value is more than 10, and CPU resources are also consumed seriously, which is caused by slow response to the site or a long time not responding to the main reason, and the system resource consumption is mainly based on the user process consumption of serious resources.
cause Analysis
The top command found that each Apache child process consumes nearly 6~8mb of memory, which is not normal. According to experience, under normal circumstances each Apache sub-process consumes about 1MB of memory, combined with Apache output log found that the site home page access frequency is the highest, that is, the home code may have problems. So check the homepage of the PHP code, found the page is very large, a lot of pictures, and by the full dynamic program composition, so that every time users visit the first page to query the database multiple times, and query database is a very CPU-intensive process, and the home PHP code does not have the corresponding caching mechanism, Each user request to re-database query operation, database query load is conceivable.
Handling Measures
Modify the homepage PHP code, reduce the page size, and increase the caching mechanism for frequently accessed operations, minimizing the program's access to the database.
4. Second analysis optimization
With the simple optimization of the front, there is a lot less downtime for system services, but occasionally the site will not be able to access it at peak times. This time still from the analysis system resource usage situation, found that the system memory resource consumption is too large, and the disk I/O has a waiting problem, and then came to the following conclusions:
cause Analysis
Memory consumption is too large, it is certain that the number of user access process is too high, before the optimization of PHP code, each Apache child process consumes 6~8MB memory, if the maximum number of users to set Apache 1024, then memory exhaustion is inevitable, when the physical memory is exhausted, virtual memory is enabled, With frequent use of virtual memory, there is a certain disk I/O wait problem that eventually results in CPU resource exhaustion.
Handling Measures
By optimizing the PHP code above, each Apache child process consumes approximately 1~2mb of memory resources, so modifying the "maxclients" option value in the Apache configuration file httpd.conf is "600" while the Apache configuration " KeepAlive "feature is turned off so that the number of Apache processes is substantially reduced, basically maintained between 500~600, although virtual memory is occasionally used, but Web services are normal and service outages are rare.
5. Third-time analysis optimization
After the first two optimizations, the site basically run normally, but at the peak of the visit occasionally there will be site unreachable phenomenon, continue to analyze the problem, through the command to view system resources, the discovery is still CPU resource exhaustion caused, but with the previous two are different:
cause Analysis
Through the observation of the background log, the PHP program has frequent access to the database operations, a large number of SQL statements have where, order by and other clauses; At the same time, too many database queries, mostly complex queries, generally need to traverse the whole table, and a large number of tables are not indexed, This program code causes the MySQL database to load too much, and the MySQL database and Apache are deployed on the same server, which causes the server to consume too much CPU resources.
Handling Measures
The SQL statements in the optimizer, increasing the matching conditions on the WHERE clause, reducing the traversal of all queries, indexing on the fields of the Where and order by clauses, and increasing the program caching mechanism, through this optimization, the website is running basically in a normal state, and there is no more downtime.

6. Fourth optimization analysis
through the previous three optimization, the site in the program code, operating system, Apache and other aspects of the optimization of space is getting smaller, to avoid the phenomenon of service gas outage, and to ensure that the site stable, efficient and fast operation, can be optimized from the structure of the website, That is, separating the Web and database deployment, you can add a dedicated database server and deploy the MySQL database separately. With the increase in traffic, if the front-end can not meet the access request, but also to increase the number of Web servers, the Web server load balanced deployment, to solve the front-end performance bottleneck, if there is still read and write pressure on the database side, you can continue to add a MySQL server, MySQL for read and write separation deployment, Such a high-performance, high-reliability Web site system is built up.

Second,   site optimization case based on dynamic, static content

1. Web site Operating Environment Description
Hardware environment: Two IBM X3850 servers, single dual core Xeon 3.0G CPU,4GB memory, 3 72GB SCSI disks.
Operating system: CentOS5.4.
Web site Architecture: Web applications are e-commerce applications based on the Java EE architecture, the Web-side application server is tomcat, the MySQL database, the Web and the database are deployed independently on both servers.

2. Performance problems and handling measures
    Description
Web pages cannot be opened when the Web page is busy, and after restarting the Java service, the site can run for a period of time, but after a while it becomes slow to respond and the final page cannot be opened.
    Check configuration
First checks the status of the system resources, discovers that the service fails, the system load is very high, the CPU runs at full load, the Java process consumes 99% of the CPU resources of the system, but the memory resources are not occupied; Then check the application server information. found that only one tomcat was running the Java program, and then looked at the Tomcat configuration file Server.xml and found that the parameters in the Server.xml file were the default configurations, without any optimizations. The default parameters of the
    actions
Server.xml file need to be modified according to the characteristics of the app, such as the ability to modify "ConnectionTimeout", "maxkeepaliverequests", " Maxprocessors "and so on several tmcat configuration file parameters, appropriate increase these several parameter values. After modifying the parameter values, continue to observe that the site service outage time interval is longer, not as frequent as before, but the Java process consumes CPU resources is very serious, Web access is very slow.

3. First analysis optimization
Since the Java process consumes serious CPU resources, you need to see what is causing the Java consumption of resources serious, through the lsof, netstat command found a large number of Java request waiting information, and then view the Tomcat log, found a large number of error messages, The log prompt and the database connection timed out, eventually unable to connect to the database, and access to the site static resources, but also inaccessible, and concluded that:
    Cause analysis
Tomcat itself is a Java container, is to use the connection/ The threading model handles the business requests, mainly for the processing of JSP, servlet and other dynamic applications, although it can also be used as an HTTP server, but the efficiency of processing static resources is very low, far less than Apache or Nginx. From the previous observation of the phenomenon, it can be preliminarily judged that Tomcat was unable to respond to client requests in a timely manner, resulting in more requests queuing until Tomcat crashed completely. For a normal access request, after the server receives the request, it gives the request to Tomcat to process, Tomcat then performs the compilation, accesses the database, and then returns the information to the client, and after the client receives the message, Tomcat closes the request link. Such a complete access process is over. And in high concurrent access, a lot of requests are instantly given to Tomcat, so that Tomcat has not finished the first request, the second request comes, then the third, and so on, so that the more accumulate, Tomcat eventually loses the response, the Java process will be in a zombie state, the resources can not be released, This is the root cause.
    Actions
to optimize Tomcat performance, you need to refactor from the fabric, first, by adding Apache support, processing static resources by Apache, processing dynamic requests by Tomcat, Use the MOD_JK module for communication between the Apache server and the Tomcat server. The advantage of using the MOD_JK module is that it can define detailed resource processing rules, and according to the characteristics of the dynamic, static Web site, the static resource files are given to Apache processing, and the dynamic request through the MOD_JK module passed to Tomcat to deal with, through the apache+jk+ The integration of Tomcat can dramatically improve the performance of Tomcat applications.

4. Second analysis optimization
After the previous optimization measures, Java resources occasionally increased, but after a period of time will be automatically reduced, this is a normal state, and in the case of high concurrent access, the Java process sometimes can not decline the resource rise, by looking at the Tomcat log, the comprehensive analysis to draw the following conclusions:
To achieve higher and more stable performance, a single Tomcat application server can sometimes fail to meet the requirements, so it is possible to run Tomcat-based load balancing systems in conjunction with the MOD_JK module, so that the front end is handled by Apache for user request scheduling, Back-end multiple tomcat is responsible for dynamic application parsing operations, the overall performance of the website will be improved by assigning the load to multiple tomcat servers.

Performance analysis and optimization case based on Web application