Use nginx + Tomcat + memcached to build Web server load balancing

1 cause

Recently, a stress test was conducted on the newly developed web system, and it was found that the response speed of concurrent logon home pages under Tomcat's default configuration pressure to 600 people was severely affected, more than 2000 errors of 500 and 502 occur in one round. I made a look at the logon time statistics and printed out the total server processing time. I saw that some responses were indeed within 20 seconds, however, the average time is far different from that tested by LR. Therefore, it can be concluded that it does not take so much time to process the program. Due to the LAN testing, the network problem can also be ruled out. This outlines the problem in the Tomcat request response capability. First, increase the number of Tomcat threads to 1000, and find that the number of errors 500 and 502 has dropped to dozens, but the response time has not improved yet. Later, two Tomcat servers were started, and nginx was used for load balancing. The response time dropped by 40%. The processing duration of the two Tomcat servers was kept at about 1 second.

It seems that Tomcat performance is indeed a bottleneck of the system. It is necessary to assume that multiple servers are used to enhance the response capability. Previously, because it was just a test logon, multiple Tomcat servers do not need to share sessions, but they must work together in real use. Record the installation and configuration process of Server Load balancer.

 
2. Solution Selection

There are several ways to work together with multiple Tomcat servers. The following solutions can be considered:

1. Use the cluster method that comes with Tomcat. For more information about tomcat, see automatically copy session information in real time. It is easy to configure. However, the efficiency of this solution is relatively low, and the performance is not good under high concurrency.

2. Use nginx's access IP-based hash routing policy to ensure that the accessed IP address is always routed to the same tomcat. This configuration is simpler. However, our application is probably because a large number of users log on to the local area network at the same time, So load balancing will be useless.

3. memcached is used to centrally manage sessions of multiple Tomcat servers. This is the most direct solution, but the operation is also the most complicated.

Our system requires both performance and better use of Server Load balancer, so 3rd solutions are the first choice. The next step is the path to installation and setup.

 
3. install and configure
3.1 install memcached

1. Download the source package of libevent-1.4.14b-stable.tar.gzand memcached-1.4.7.tar.gz. The former is the dependency package of the latter and is an event-driven package.

2) the installation is very smooth, or the classic compilation and installation commands:

Tar zxvf libevent-1.4.14b-stable.tar.gz
CD libevent-1.4.14b-stable
../Configure -- prefix =/usr/local/libevent-1.4.14b
Make
Make install

Tar zxvf memcached-1.4.7.tar.gz
CD memcached-1.4.7
./Configure -- prefix =/usr/local/memcached-1.4.7 -- With-libevent =/usr/local/libevent-1.4.14b/
Make
Make install

3) Start memcached:

./Bin/memcached-D-M 256-u root-P 11211-C 1024-P/tmp/memcached. PID

3.2 memcached-session-Manager Configuration

It is already a mature solution for Tomcat to call memcached to store sessions. The open-source MSM can solve this problem. The jar package is used, which is vague in the official documentation. Here I use the kryo serialization solution, so I use more packages:

Kryo-1.03.jar

Kryo-serializers-0.8.jar

Memcached-2.5.jar (I officially see the latest has reached 2.7, but the MSM official said with 2.5, may not test the new package, especially the version of changelog 2.6 mentioned in the API has been adjusted, or don't upgrade it)

Memcached-session-manager-1.5.1.jar

Memcached-session-manager-tc7-1.5.1.jar

Minlog-1.2.jar

Msm-kryo-serializer-1.5.1.jar

Reflectasm-0.9.jar

These packages are all stored in the $ catalina_home/lib directory.

In addition, the four serialization solutions officially provided, kryo is the most efficient. For more information, see http://code.google.com/p/memcached-session-manager/wiki/serializationstrategies.

 

Next, modify the configuration file $ catalina_home/CONF/context. xml of Tomcat and change it to the new session storage method. Add the following content to the configuration file:

 
 

  
  
    <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager" 
  
  
      memcachedNodes="n1:127.0.0.1:11211" 
  
  
      sticky="false" 
  
  
      lockingMode="auto" 
  
  
      sessionBackupAsync="false"   
  
  
      sessionBackupTimeout="1000"  
  
  
 transcoderFactoryClass="de.javakaffee.web.msm.serializer.kryo.KryoTranscoderFactory" 
  
  
      /> 
 
 

 

Add de. javakaffee. Web. MSM. Level = fine to the $ catalina_home/CONF/logging. properties file to view detailed Session access information in the Catalina. Out log.

 

In addition, add requesturiignorepattern = "in the Manager configuration ". *\. (PNG | GIF | JPG | CSS | JS) $ ". In Chrome browser test, the sessionid is suddenly changed, and then the interceptor jumps back to the homepage. If it is removed, everything will be normal, but the interceptor will only detect the action. It should be reasonable to say that it doesn't matter at all. Hope you can give me some advice!

 

3.3 nginx Configuration

Nginx is very simple. You only need to configure several more servers in upstream. Here I will post my Configuration:

 
 

  
  
# User nobody;
  
  
Worker_processes 16;
  
  
 
  
  
 
  
  
Events {
  
  
Use epoll;
  
  
Worker_connections 65535;
  
  
}
  
  
 
  
  
 
  
  
HTTP {
  
  
Include mime. types;
  
  
Default_type application/octet-stream;
  
  
 
  
  
# Log_format main '$ remote_addr-$ remote_user [$ time_local] "$ request "'
  
  
# '$ Status $ body_bytes_sent "$ http_referer "'
  
  
# '"$ Http_user_agent" "$ http_x_forwarded_for "';
  
  
 
  
  
# Access_log logs/access. Log main;
  
  
 
  
  
Client_header_buffer_size 32 K;
  
  
Large_client_header_buffers 4 32 K;
  
  
Client_max_body_size 8 m;
  
  
Client_body_buffer_size 128 K;
  
  
 
  
  
Sendfile on;
  
  
Tcp_nopush on;
  
  
 
  
  
# Keepalive_timeout 0;
  
  
Keepalive_timeout 65;
  
  
 
  
  
Gzip on;
  
  
Gzip_types text/JavaScript text/plain text/CSS application/XML application/X-JavaScript;
  
  
Gzip_disable "MSIE [1-6] \. (?!. * Sv1 )";
  
  
 
  
  
Proxy_connect_timeout 300;
  
  
Proxy_send_timeouts 300;
  
  
Proxy_read_timeout 300;
  
  
Proxy_buffer_size 16 K;
  
  
Proxy_buffers 4 32 K;
  
  
 
  
  
Proxy_set_header X-forwarded-for $ remote_addr;
  
  
Proxy_set_header connection close;
  
  
Server_names_hash_max_size 1024;
  
  
Server_names_hash_bucket_size 1024;
  
  
 
  
  
# Default cache parameters for use by Virtual Hosts
  
  
# Set the cache path to tmpfs mounted disk, and the zone name
  
  
# Set the maximum size of the on disk cache to less than the tmpfs file system size
  
  
Proxy_cache_path./cache levels = keys_zone = pscms: 100 m max_size = 800 m;
  
  
Proxy_temp_path./proxy;
  
  
 
  
  
# Configure backend server information
  
  
Upstream web_server {
  
  
# Ip_hash;
  
  
Server localhost: 8080 max_fails = 3 fail_timeout = 30 s;
  
  
Server localhost: 8180 max_fails = 3 fail_timeout = 30 s;
  
  
}
  
  
 
  
  
Server {
  
  
Listen 8888; # Listen for IPv4
  
  
# Listen [:]: 80 default ipv6only = on; # Listen for IPv6
  
  
SERVER_NAME localhost;
  
  
 
  
  
Charset UTF-8;
  
  
Log_format main '$ remote_addr-$ remote_user [$ time_local] "$ request "'
  
  
'$ Status $ body_bytes_sent "$ http_referer "'
  
  
'"$ Http_user_agent" "$ http_x_forwarded_for "';
  
  
Access_log logs/host. Access. Log main;
  
  
# Access_log off;
  
  
 
  
  
Location ~ . * \. (JSP | action )? $ {
  
  
Proxy_set_header host $ http_host;
  
  
Proxy_redirect off;
  
  
Proxy_pass http: // web_server;
  
  
Proxy_set_header host $ host;
  
  
Proxy_set_header X-real-IP $ remote_addr;
  
  
Proxy_set_header X-forwarded-for $ proxy_add_x_forwarded_for;
  
  
}
  
  
 
  
  
Location ~ . * \. (GIF | JPG | JPEG | PNG | BMP | SWF | JS | CSS) $ {
  
  
# If the backend server returns errors such as 502, 504, and execution timeout, the request is automatically forwarded to another server in the upstream Server Load balancer pool for failover.
  
  
Proxy_next_upstream http_502 http_504 error timeout invalid_header;
  
  
 
  
  
Proxy_cache pscms; # cache, using the Web cache zone cache_one
  
  
Proxy_cache_valid 200 304 1 h; # set different cache times for different HTTP Status Codes
  
  
Proxy_cache_valid 301 302 5 m;
  
  
Proxy_cache_valid any 1 m;
  
  
Proxy_set_header host $ host;
  
  
Proxy_set_header X-real-IP $ remote_addr;
  
  
Proxy_set_header X-forwarded-for $ remote_addr;
  
  
Proxy_set_header accept-encoding "; # (or the backend server closes gzip), so that this machine will not cache compressed files, causing garbled characters
  
  
Proxy_ignore_headers "cache-control" "expires"; # after this configuration is added, proxy_cache supports the expires set in the background.
  
  
Proxy_pass http: // web_server;
  
  
Expires 15 m;
  
  
}
  
  
 
  
  
Location /{
  
  
Proxy_set_header host $ http_host;
  
  
Proxy_redirect off;
  
  
Proxy_pass http: // web_server;
  
  
Proxy_set_header host $ host;
  
  
Proxy_set_header X-real-IP $ remote_addr;
  
  
Proxy_set_header X-forwarded-for $ proxy_add_x_forwarded_for;
  
  
}
  
  
 
  
  
}
  
  
 
  
  
}
 
 

 

Reference:

1. http://code.google.com/p/memcached-session-manager/wiki/SetupAndConfiguration

2. http://wangrui.iteye.com/blog/500921

3 http://passover.blog.51cto.com/2431658/648182