Apache + Tomcat + MySQL Server Load balancer and clusters

Preface:
The company has developed a website. The estimated maximum number of online users is 30 thousand, and the maximum number of concurrent users is 100. Whether or not the developed website can withstand this pressure and how to ensure that the load of the website is no problem, after research, it is decided as follows:
(1) Server Load balancer and cluster technologies are adopted, and Apache + Tomcat cluster technologies are initially adopted.
(2) Use stress testing tools to test the pressure. The tool is LoadRunner.
Hardware environment construction:
To be able to perform stress testing, we need to build an environment. At the beginning, the test was carried out in the company's Lan, but soon a problem was found, that is, the stress test results of a script were different each time, and the difference was great. It was originally influenced by the company's network, so we decided to build a fully isolated LAN test. The LAN configuration is as follows:
(1) network speed: 100 m
(2) three servers:
Server Load balancer: Windows2003,
Tomcat server: Windows2000 Professional
Database Server: Operating System Windows2000 Professional
The CPU usage of the three machines is 2.4 GB, and the memory is 1 GB.
Software environment construction:
The software version is as follows:
Apache version: 2.054,
Tomcat5.0.30,
MySQL: 4.1.14.
Jdk1.5
Stress Testing Tool: loadrunner7.8.

The load balancing solution is as follows:
One server (Operating System 2003) installs Apache as the load server and tomcat as a worker. One server installs Tomcat as the second worker, and the other server acts as the database server.
The load balancing between APACHE and tomcat adopts jk1.2.14 (2.0 is not used, mainly because 2.0 is no longer maintained ).
Cluster solution:
Use the cluster solution of Tomcat itself. Configure in server. xml.
Stress testing problems:
After stress testing, I found some problems, which are listed one by one:
(1) After the tocmat cluster is used, the speed becomes very slow. After the cluster, session replication is required, resulting in slow speed. Tomcatd replication, currently does not support application replication. The role of replication is mainly used for fault tolerance, that is, after a machine fails, Apache can automatically forward requests to another machine. In consideration of fault tolerance and speed, we finally chose the speed and removed the Tomcat cluster.
(2) Restrictions on the maximum number of concurrent users in the operating system:
In order to use the website pressure, we started to test only the maximum load of Tomcat. The operating system installed on the Tomcat server is Windows2000 Professional. When we use a stress testing tool for concurrent testing, we find that as long as there are more than 15 concurrent users, we often cannot connect to the server. After research, it is found that the problem is the operating system: Windows professional supports a limited number of concurrent users, the default seems to be 15. Therefore, we use all the operating systems of Windows2003 server.
(3) database connection pool problems:
When testing the database connectivity performance, we found that the database connection speed was slow. The connection performance is much worse with the addition of some users. The database connection pool we use is DBCP, and the default Initialization is 50. It should not be very slow. Queries the number of connections to the database. Only one connection is initialized. When a user is added concurrently, the program creates a new connection, resulting in a slow connection. The reason is here. How can this problem be solved? Occasionally, the database connection pressure test is executed under tomcat5.0.30 under jdk1.4. It is found that the speed of database connection creation is very fast. It seems that the JDBC driver of jdk1.5 has a problem. So we changed the JDK version to 1.4.

(4) c3p0 and DBCP
C3p0 is the default built-in database connection pool of hibernate3.0, and DBCP is the database connection pool developed by Apache. We tested and compared the two connection pools, and found that DBCP was about 1 second faster than the average time of c3p0 when the number of concurrent users was less than 300. However, when the number of concurrent users is 400, the two are similar.

Although DBCP is faster than c3p0 in terms of speed, there is a bug: When the database connection established by DBCP is broken for some reason, DBCP will no longer create a new connection, as a result, you must restart tomcat to solve the problem. DBCP bug makes us decide to use c3p0 as the database connection pool.
Adjusted scheme:
Windows Server
Jdk1.4
Tomcat 5.0.30
Database Connection Pool c3p0
Use Load Balancing only, not Cluster .
Software Configuration:
Apache configuration: Mainly configures httpd. conf and the newly added file workers. properties.
Httpd. conf:
# Maximum number of requests for a connection
Maxkeepaliverequests 10000
# NT environment, only this parameter can be configured to provide performance
<Ifmodule mpm_winnt.c>
# The number of threads for each process. The maximum value is 1920. NT only starts two parent and child processes, and cannot start multiple processes
Threadsperchild 1900
Maximum number of requests that each sub-process can process
Maxrequestsperchild 10000
</Ifmodule>

# Load mod_jk
#
Loadmodule jk_module modules/mod_jk.so
#
# Configure mod_jk
#
Jkworkersfile CONF/workers. Properties
Jklogfile logs/mod_jk.log
Jkloglevel info
# Request Distribution: Send JSP files,. Do and other dynamic requests to Tomcat for processing
DocumentRoot "C:/Apache/htdocs"
Jkmount/*. jsp loadbalancer
Jkmount/*. Do loadbalancer
Jkmount/servlet/* loadbalancer
# Turn off the host lookup. If it is on, it will affect performance and may have a latency of more than 10 seconds.
Hostnamelookups off
# Cache Configuration
Loadmodule cache_module modules/mod_cache.so
Loadmodule disk_cache_module modules/mod_disk_cache.so
Loadmodule mem_cache_module modules/mod_mem_cache.so

<Ifmodule mod_cache.c>
Cacheforcecompletion 100
Cachedefaultexpire 3600
Cachemaxexpire 86400
Cachelastmodifiedfactor 0.1

<Ifmodule mod_disk_cache.c>
Cacheenable Disk/
Cacheroot C:/cacheroot
Cachesize 327680
Cachedirlength 4
Cachedirlevels 5
Cachegcinterval 4
</Ifmodule>
<Ifmodule mod_mem_cache.c>
Cacheenable MEM/
Mcachesize 8192
Mcachemaxobjectcount 10000
Mcacheminobjectsize 1
Mcachemaxobjectsize 51200
</Ifmodule>
</Ifmodule>
Worker. properties File
#
# Workers. properties, refer
Http://jakarta.apache.org/tomcat/connectors-doc/config/workers.html
# In UNIX, we use forward slashes:
PS =

# List the workers by name

Worker. List = worker at1, tomcat2, loadbalancer

#------------------------
# First tomcat server
#------------------------
Worker. tomcat1.port = 8009
Worker. tomcat1.host = localhost
Worker. tomcat1.type = ajp13

# Specify the size of the open connection cache.
# Worker. tomcat1.cachesize

#
# Specifies the load balance factor when used
# A Load Balancing worker.
# Note:
# ----> Lbfactor must be> 0
# ----> Low lbfactor means less work done by the worker.
Worker. tomcat1.lbfactor = 900

#------------------------
# Second Tomcat server
#------------------------
Worker. tomcat1.port = 8009
Worker. tomcat1.host = 202.88.8.101
Worker. tomcat1.type = ajp13

# Specify the size of the open connection cache.
# Worker. tomcat1.cachesize

#
# Specifies the load balance factor when used
# A Load Balancing worker.
# Note:
# ----> Lbfactor must be> 0
# ----> Low lbfactor means less work done by the worker.
Worker. tomcat1.lbfactor = 2000

#------------------------
# Load balancer worker
#------------------------

#
# The loadbalancer (Type LB) worker performs Weighted Round-Robin
# Load Balancing with sticky sessions.
# Note:
# ----> If a worker dies, the Load balancer will check its state
# Once in a while. Until then all work is redirected to peer
# Worker.
Worker. loadbalancer. type = LB
Worker. loadbalancer. balanced_workers = worker at1, tomcat2

#
# End workers. Properties
#

Tomcat1 Configuration:
<! -- Configure Server. xml
Remove port 8080 and comment out the following code: -->
<Connector
Port = "8080" maxthreads = "150" minsparethreads = "25" maxsparethreads = "75"
Enablelookups = "false" redirectport = "8443" acceptcount = "100"
DEBUG = "0" connectiontimeout = "20000"
Disableuploadtimeout = "true"/>

<! -- Configure the port 8009 as follows: -->
<Connection Port = "8009"
Maxthreads = "500" minsparethreads = "400" maxsparethreads = "450"
Enablelookups = "false" redirectport = "8443" DEBUG = "0"
Protocol = "AJP/1.3" type = "codeph" text = "/codeph"/>
<! -- Configure the engine -->
<Engine name = "Catalina" defaulthost = "localhost" DEBUG = "0" jvmroute = "tomcat1">

Start the memory configuration and develop the configure Tomcat program to configure:
Initial memory pool: 200 m
Maxinum memory pool: 300 m
Tomcat2 Configuration:
The configuration is similar to the tomcat1, and the change is as follows:
<! -- Configure the engine -->
<Engine name = "Catalina" defaulthost = "localhost" DEBUG = "0" jvmroute = "tomcat2">

Start the memory configuration and develop the configure Tomcat program to configure:
Initial memory pool: 512 m
Maxinum memory pool: 768 m
MySQL Configuration:
Server Type: dedicated MySQL server machine
Database usage: transational database only
Concurrent connections: online transaction processing (OLTP)
Character Set: utf8
Database Connection Pool Configuration:
We use the Spring framework and the configuration is as follows:
<Property name = "hibernateproperties">
<Props>
<Prop key = "hibernate. dialect"> org. hibernate. dialect. mysqldialect </prop>
<Prop key = "hibernate. Connection. driver_class"> com. MySQL. JDBC. Driver </prop>
<Prop key = "hibernate. Connection. url"> JDBC: mysql: // 202.88.1.103/DB </prop>
<Prop key = "hibernate. Connection. username"> SA </prop>
<Prop key = "hibernate. Connection. Password"> </prop>

<Prop key = "hibernate. show_ SQL"> false </prop>
<Prop key = "hibernate. use_ SQL _comments"> false </prop>

<Prop key = "hibernate. cglib. use_reflection_optimizer"> true </prop>
<Prop key = "hibernate. max_fetch_depth"> 2 </prop>

<Prop key = "hibernate. c3p0. max_size"> 200 </prop>
<Prop key = "hibernate. c3p0. min_size"> 5 </prop>
<Prop key = "hibernate. c3p0. Timeout"> 12000 </prop>
<Prop key = "hibernate. c3p0. max_statements"> 50 </prop>
<Prop key = "hibernate. c3p0. acquire_increment"> 1 </prop>
</Props>
</Property>
There are no additional configurations for others.
LoadRunner FAQs:
(1) sofeware caused connction: in this case, it is generally because the script has a problem or the LoadRunner has a problem. Solution: restart the machine or record the script. It is estimated that it is a LoadRunner bug.
(2) cannot connect to server: unable to connect to the server. This is because the configuration of the server is faulty and the server cannot withstand excessive concurrent connections. You need to optimize the server configuration,
If the operating system uses Windows 2003 Server,
Optimize tomcat configuration: maxthreads = "500" minsparethreads = "400" maxsparethreads = "450 ". However, Tomcat supports up to 500 concurrent accesses.
Optimize Apache configuration:
Threadsperchild 1900
Maxrequestsperchild 10000
Other errors include:
Action. C (10): Error-27791: Server has shut down the connection prematurely
HTTP status-code = 503 (service temporarily unavailable)
Generally, it is caused by insufficient server configurations. Handle the problem according to Problem (2). If the problem persists, optimize the hardware and adjust the program.
Apache problems:
(1) file does not exist: C:/Apache/htdocs/favicon. ICO:
This problem is caused by the absence of the favicon. ICO file in the Apache and htdocs directories. This file is a website icon and only appears in Firefox, myie, and other browsers.
(2) images cannot be displayed:
After Apache is configured, Images cannot be displayed.
Solution: copy the program image to the Apache htdocs directory according to the program structure.
(3) unable to process the request:
After the ***. Do command is input, Apache returns an error message, but Tomcat connection is normal. The reason is that the. Do command is not forwarded to Tomcat for processing. The solution is as follows:
Configure the following content in the Apache configuration file:
DocumentRoot "C:/Apache/htdocs"
Jkmount/*. jsp loadbalancer
Jkmount/*. Do loadbalancer

Summary:
Website Stress Testing It involves a wide range of knowledge, not only familiar with stress testing tools, but also how Configuration And optimize the application server and database, and need to know how to optimize the network, operating system, hardware system.
In the test, you must be good at discovering problems and know how to solve them. The most important thing is to have a good test method. At the beginning of the test, you can start with the simplest test script without complicated scripts, so that you can easily find problems. For example, at the beginning, we started with a simple script for downloading the login interface to test the pressure of a tomcat Load . A simple login script helps us optimize tomcat configuration. Then we test the database connection, which is also a simple database connection script that helps us optimize Database Connection Pool And then use these simple scripts to test Apache load balancing and optimize Apache configuration. Finally, run complex scripts to simulate the processing of users with multiple roles at different times to test the website load.
 

 

DBCP uses the object pool objectpool of Apache as the implementation of the connection pool. when constructing the genericobjectpool, an embedded class Evictor is generated to implement the self-runnable interface. If _ timeout is greater than 0, every _ timebetweenevictionrunsmillis millisecond Evictor calls the evict () method to check whether the idle time of the object is greater than _ minevictableidletimemillis Millisecond (_ minevictableidletimemillis is less, the default value is 30 minutes. If yes, the object is destroyed. Otherwise, the object is activated and verified. Then, the ensureminidle method is called to check whether the number of objects in the pool is no less than _ minidle. After calling the returnobject method to put the object back into the object pool, first check whether the object is valid, and then call the passivateobject method of poolableobjectfactory to make the object inactive. Check whether the number of objects in the object pool is smaller than _ maxidle. If yes, you can put the object back into the object pool. Otherwise, the object will be destroyed.

There are also several important attributes, including _ testonborrow, _ testonreturn, and _ testwhileidle. These attributes indicate whether to verify the retrieved, returned, and idle objects and check whether the objects are valid, the default value is false. Therefore, when DBCP is used, the database connection is disconnected for some reason, and the connection is obtained from the connection pool without verification. In this case, the database connection is invalid when the connection is actually obtained. This is the case for many DBCP bugs on the Internet. Only by setting these attributes to true and then providing the _ validationquery statement can ensure that the database connection is always effective, oracle databases can use select count (*) from dual. However, DBCP requires that the record set queried by the _ validationquery statement must not be empty. This may be a small bug, in fact, you only need to run the _ validationquery statement.