Nginx+tomcat+memcached High-availability session hold

I. Overview

The previous article has described the enterprise high-availability load-related architecture and implementation, which commonly used nginx or Haproxy,lvs combined with keepalived to do the front-end high-availability scheduler, but did not mention that the session is highly available to maintain;
This article implements the Tomcat Memcache session server high-availability sessions cache service through Tomcat session Replication Cluster (Tomcat band) and Tomcat binding Memcat and third party components;
The effect of the implementation:
The same client access to the business site, through the scheduler load dispatch to the backend, regardless of the choice of the back end, the session ID is unchanged, are stored in two or more memcached cache (load redundancy), to maintain the session;

Architecture diagram:

Description: When the client requests, Nginx dispatches the request to a back-end server through the load scheduling algorithm, and Tomcat replicates the conversation through multicast to the nodes of the cluster; All nodes share the session;

Description: When the client requests, the request is dispatched to a back-end server by the load scheduling algorithm, and the session is stored in two memcached; when the client refreshes (without changing the browser), the session ID remains the same when the request is replaced by another backend server;

Test environment:
Nginx:centos7 Epel installation Nginx wan:172.16.3.152 lan:192.168.10.254
Tomcat A:CENTOS7 node1.san.com Epel installing tomcat 7 openjdk-1.8.0 memcached (in real-world standalone server)
Tomcat B:CENTOS7 nodde2.san.com Epel install tomcat 7 openjdk-1.8.0 memcached a standalone server in the real world)
Test client Ubuntu 16.04

cat /etc/hosts172.16.3.152       www.san.com

Second, install the configuration cluster

Nginx Installation

[[email protected] ~]# yum install epel-release -y[[email protected] ~]# yum install nginx -y

Nginx Configuration
Add the following line in the/etc/nginx/nginx.conf HTTP segment

    upstream tcsrvs {        server 192.168.10.11:8080;        server 192.168.10.12:8080;        }

Cat/etc/nginx/conf.d/san.com.conf

[[email protected] ~]# cat /etc/nginx/conf.d/san.com.conf server {            listen 80;            server_name www.san.com;            location / {                proxy_pass http://tcsrvs;                }        }

Tomcat configuration:
Both units need to be installed

#yum install epel-release -y#yum install java-1.8.0 java-1.8.0-openjdk-devel tomcat tomcat-webapps tomcat-admin-webapps tomcat-docs-webapp  -y

Description: You can also download tomcat to the specified directory and add environment variables via Oracle's official download JDK; In order to be quick, we use the stable version in the Epel warehouse;

Add a test page
Yum installed Tomcat working directory in/var/lib/tomcat/webapps on the Node1 and Node2, in this directory to create a test project, the title is called Tomcat A and Tomcat B color respectively green and red, to show the difference; Production environment Node1 and node2 content; here in order to test the difference between Node1 and node2 content;

#mkdir -pv /var/lib/tomcat/webapps/test/{WEB-INF,META-INF,classes,lib}#cat /var/lib/tomcat/webapps/test/index.jsp<%@ page language="java" %> 

Configure Administration page Password
Configure access password after Tomcat and hypervisor are installed
Modify a comment/etc/tomcat/tomcat-users.xml file

<role rolename="admin-gui"/><role rolename="manager-gui"/><user username="tomcat" password="tomcat" roles="manager-gui,admin-gui"/>

Backing up the default/etc/tomcat/server.xml file

cd /etc/tomcatcp server.xml server.xml_def

Test page Access
Http://www.san.com/test: Tomcat A appears

Ctrl+f5 forced Flush appears tomcat B


Issue: If it is a two-content configuration, client access is refreshed and switched to another backend processing; how do you keep a service that retains information through the session (purchase cart)? In other words, how to keep the session uninterrupted, regardless of whether the request is assigned to that backend?

Solution Solutions
1) session sticky (sticky): Divided into source_ip based on source IP and cookie
The SOURCE_IP has different implementations on different schedulers:
Lvs:sh algorithm;
Nginx:ip_hash or hash $request _uri consistent (consistent hashing algorithm)
Haproxy:source

Cookies:
Nginx:hash or hash $cookie _name consistent;
Haproxy:cookie

2) Conversation cluster (session cluster):D ELTA Session Manager
3) session Server:redis (store), memcached (cache)

The following is a session hold function based on the Tomcat self-brought session cluster and memcached;

Third, Tomcat Session Replication cluster configuration

Tomcat session Replication Cluster Chinese is also called Tomcat sessions replication cluster, that is, the session is replicated to each backend Tomcat node by means of multicast;
Refer to your own Help documentation: http://www.san.com/docs/cluster-howto.html
The two Node1 node2 node/etc/hosts are added as follows:

#cat /etc/hosts192.168.10.11 node1.san.com node1192.168.10.12 node2.san.com node2

Add the following in the host field of the two Tomcat node Sever.xml:

<cluster classname= "Org.apache.catalina.ha.tcp.SimpleTcpCluster" channelsendoptions= "8" > &L T                   Manager classname= "Org.apache.catalina.ha.session.DeltaManager" expiresessionsonshutdown= "false" Notifylistenersonreplication= "true"/> <channel classname= "Org.apache.catalina.tribes.group.Group                        Channel "> <membership classname=" Org.apache.catalina.tribes.membership.McastService "                        address= "228.10.0.4" port= "45564" frequency= "500"                      Droptime= "/> <receiver classname=" Org.apache.catalina.tribes.transport.nio.NioReceiver "                      address= "Auto" <!--requires native IP---port= "4000" if no/etc/hosts parsing Autobind= "selectortimeout=" maxthreads= "6"/> &L T Sender classname= "orG.apache.catalina.tribes.transport.replicationtransmitter "> <transport classname=" org.apache.catalina.t Ribes.transport.nio.PooledParallelSender "/> </Sender> <interceptor classname=" Org.apache . Catalina.tribes.group.interceptors.TcpFailureDetector "/> <interceptor classname=" Org.apache.catalina.tri Bes.group.interceptors.MessageDispatch15Interceptor "/> </Channel> <valve classname=" Org.apac He.catalina.ha.tcp.ReplicationValve "filter=" "/> <valve classname=" Org.apache.catalina.ha.s Ession.                    Jvmroutebindervalve "/> <deployer classname=" Org.apache.catalina.ha.deploy.FarmWarDeployer "                     Tempdir= "/tmp/war-temp/" deploydir= "/tmp/war-deploy/" watchdir= "/tmp/war-listen/" Watchenabled= "false"/> <clusterlistener classname= "Org.apache.catalina.ha.session.JvmRo UtesessionidbindeRlistener "/> <clusterlistener classname=" Org.apache.catalina.ha.session.ClusterSessionListener "/> </Cluster>

Under Copy/etc/tomcat/web.xml/var/lib/tomcat/webapps/test/web-inf/and add "<distributable/" under the "<web-app >" field in Web. xml > ";
Restart Tomcat and visit Http://www.san.com/test again:

Ctrl + F5 Force refresh:

A session can be persisted, as long as it is requested from the same client, refreshed or closed again (based on the same browser) as long as the session is not expired, the session ID is the same regardless of the backend;

Disadvantages:
Tomcat comes with a support session cluster (can send various nodes in multicast mode); But there is one drawback; the back-end tomcat node is inefficient when it is too large;

Iv. Tomcat Memcache Session Server high-availability configuration

Principle Description:
When a client request arrives at the front-end Nginx Scheduler and is assigned to a Tomcat node on the backend, Tomcat takes precedence over the native memory save session, and when a request is finished, Tomcat passes through the third-party component (Kryo,javolution,xstream, Flexjson) The session is serialized and sent to the memcached node for backup, the second request, if the local session is returned directly, the second request ends, the session modified information to update to the backend memcached server, In this way, keep the local session synchronized with the session on the memcached. When the Tomcat node goes down, the user's next request is routed to the other Tomcat node by the front-end load balancer, which does not have the user's session information, which reads the session from the memcached server. and save the session to the local memory, when the request is over, the session is modified, and then sent back to memcached for storage backup
When more than one memcached is configured on the backend, Tomcat updates the session to multiple memcached nodes at the same time when the session information is updated, and when a memcached node fails, Tomcat can select a working memcached node to read the session information to send to the user's browser, let it reset session information, so that memcached also reached the purpose of high availability;
The following operations are all operated on two node
Restore the default configuration file

#cd /etc/tomcat/#cp server.xml server.xml_cluster#cp server.xml_def server.xml#systemctl stop tomcat

Installing the Memcached Service

#yum install memcached -y#systemctl start memcached

Memcache configuration (default, increased memory and number of concurrent connections required for production environments)

# cat /etc/sysconfig/memcached PORT="11211"USER="memcached"MAXCONN="1024"CACHESIZE="64"OPTIONS=""

Add the following to the two/etc/tomcat/server.xml host segments:

<Context path="/test" docBase="test" reloadable="true">        <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"                memcachedNodes="m1:192.168.10.11:11211,m2:192.168.10.12:11211"                failoverNodes="m1"                requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"                transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory"/>        </Context>

Description
Add two redundant backup memcached nodes are called m1,m2 failovernodes= "M1" means M1 as backup, when M2 fails, even with M2;

Install the corresponding version component

下载以下JAR包到tomcat库目录;cd /usr/share/tomcat/lib wget http://www.java2s.com/Code/JarDownload/javolution/javolution-5.5.1.jar.zip  #需要解压 unzip javolution-5.5.1.jar.zipwget http://repo1.maven.org/maven2/net/spy/spymemcached/2.12.1/spymemcached-2.12.1.jarwget http://repo1.maven.org/maven2/de/javakaffee/msm/msm-javolution-serializer/2.1.1/msm-javolution-serializer-2.1.1.jarwget http://repo1.maven.org/maven2/de/javakaffee/msm/memcached-session-manager-tc7/2.1.1/memcached-session-manager-tc7-2.1.1.jarwget http://repo1.maven.org/maven2/de/javakaffee/msm/memcached-session-manager/2.1.1/memcached-session-manager-2.1.1.jar

Note: Epel installs the Tomcat and OPENJDK versions as follows:
OPENJDK: "1.8.0_161"
Tomcat: "7.0.76"
The above third-party plug-ins must be compatible with the corresponding version, if there is a problem with Tomcat boot, cannot access or similar errors as follows

#tail -fn 100 /var/log/tomcat/catalina.xxxx.log三月 23, 2018 4:12:52 下午 org.apache.catalina.core.StandardContext startInternal严重: The session manager failed to startorg.apache.catalina.LifecycleException: Failed to start component [de.javakaffee.web.msm.MemcachedBackupSessionManager[/test]]    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:162)    at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5643)    at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)

Indicates that the third-party component is incompatible with Tomcat! Please re-download the version;

Test:
Browser Access http://www.san.com/test:

Ctrl+f5 Strong refresh:

From the test, it can be seen that cache information such as session is stored by memcache, and synchronized to two memcache; only m2 nodes are currently used;

Summarize:

Fast implementation of the load Tomcat application via Nginx; The reference session is not a problem; The session is maintained through the use of the Tomcat session Replication cluster and the combination of memcached and third-party components for high-availability session caching services; The former is not suitable for large-scale application;

Nginx+tomcat+memcached High-availability session hold