Lvs+keepalived+nginx implementation of high performance load Balancing cluster high performance JSP cluster

lvs-master:192.168.254.134
lvs-backup:192.168.254.135
lvs-vip:192.168.254.88
nginx+tomcat:192.168.254.131
nginx+tomcat:192.168.254.132
nginx+tomcat:192.168.254.133
(multiple can be added as required)

Install the base package
Yum install-y gcc-c++ pcre pcre-devel zlib zlib-devel OpenSSL openssl-devel

Installing Nginx
(Download path for your own selection)
Download nginx1.14.0 stable version
wget http://nginx.org/download/nginx-1.14.0.tar.gz
Decompression: TAR-ZXF nginx-1.14.0.tar.gz
Enter the extracted source directory: CD nginx-1.14.0
Use the default configuration:./configure
Compiling and installing Nginx:make && make install
Start Nginx:/usr/local/nginx/sbin/nginx
Quick Stop Nginx:/usr/local/nginx/sbin/nginx-s Stop
Graceful Stop Nginx (will process the request in Process complete):/usr/local/nginx/sbin/nginx-s quit
Detect configuration Error:/usr/local/nginx/sbin/nginx-t
Reload configuration:/usr/local/nginx/sbin/nginx-s Reload
1, write Nginx startup script, and join the system services
Vim/etc/init.d/nginx
and write the following in it:

#!/bin/bash
#chkconfig:-30 21
#description: HTTP service.
#Source Function Library
. /etc/init.d/functions
#Nginx Settings
Nginx_sbin= "/usr/local/nginx/sbin/nginx"
nginx_conf= "/usr/local/nginx/conf/nginx.conf"
Nginx_pid= "/usr/local/nginx/logs/nginx.pid"
Retval=0
Prog= "Nginx"
Start () {
Echo-n $ "Starting $prog:"
Mkdir-p/dev/shm/nginx_temp
Daemon $NGINX _sbin-c $NGINX _conf
Retval=$?
Echo
Return $RETVAL
}
Stop () {
Echo-n $ "Stopping $prog:"
Killproc-p $NGINX _pid $NGINX _sbin-term
Rm-rf/dev/shm/nginx_temp
Retval=$?
Echo
Return $RETVAL
}
Reload () {
Echo-n $ "Reloading $prog:"
Killproc-p $NGINX _pid $NGINX _sbin-hup
Retval=$?
Echo
Return $RETVAL
}
Restart () {
Stop
Start
}
Configtest () {
$NGINX _sbin-c $NGINX _conf-t
return 0
}
Case "$" in
Start
Start
;;
Stop
Stop
;;
Reload
Reload
;;
Restart
Restart
;;
Configtest)
Configtest
;;
*)
echo $ "Usage: $ start|stop|reload|
Restart|configtest} "
Retval=1
Esac
Exit $RETVAL

and change the file execution permissions: chmod 755/etc/init.d/nginx
Join system service Startup list: Chkconfig--add nginx
and enable Boot: Chkconfig nginx on
Open service: Nginx start or/etc/init.d/nginx Start|stop|restart
Check to see if the Nginx page is properly accessible.
http://192.168.254.131/

Installing the base Environment JDK
wget http://download.oracle.com/otn-pub/java/jdk/8u171-b11/512cd62ec5174c3487ac17c61aaa89e8/ Jdk-8u171-linux-x64.tar.gz? Authparam=1530931780_fdbf6ecf3dfcc8e6eb6fcef06a7eaaec

TAR-ZXF jdk-8u171-linux-x64.tar.gz-c/usr/local/
cd/usr/local/
MV jdk1.8.0_171/jdk1.8
Vi/etc/profile
At the end of the Add JDK environment variable
Export java_home=/usr/local/jdk1.8
Export path= $JAVA _home/bin: $PATH
Export classpath=.: $JAVA _home/lib/dt.jar: $JAVA _home/lib/tools.jar
Make environment variables effective
Source/etc/profile
View JDK version
Java-version
Java Version "1.8.0_171"
Java (TM) SE Runtime Environment (build 1.8.0_171-B11)
Java HotSpot (TM) 64-bit Server VM (build 25.171-b11, Mixed mode)
View JDK paths
Echo $JAVA _home
/usr/local/jdk1.8

Installing Tomcat
wget http://mirrors.hust.edu.cn/apache/tomcat/tomcat-9/v9.0.10/bin/apache-tomcat-9.0.10.tar.gz

TAR-ZXF apache-tomcat-9.0.10.tar.gz
MV Apache-tomcat-9.0.10/usr/local/tomcat
Create a project directory
Mkdir-p/home/www/web/(Write a index.jsp inside)
VI index.jsp (save to launch)
Modify the Tomcat access path
cd/usr/local/tomcat/conf/
VI Server.xml
Unpackwars= "true" autodeploy= "true" >
This line is added.
<context path= "" docbase= "/home/www/web" reloadable= "true"/>

See if the Tomcat page will be accessed properly
http://192.168.254.131:8080/

Configuring Nginx and Tomcat load balancing/reverse Proxies
cd/usr/local/nginx/conf/
VI nginx.conf
User www www;
Worker_processes 4;

Error_log/usr/local/nginx/logs/error.log;
Error_log/usr/local/nginx/logs/error.log notice;
Error_log/usr/local/nginx/logs/error.log info;

Pid/usr/local/nginx/logs/nginx.pid;

#工作模式及连接数上限
Events {
Use Epoll;

worker_connections  65535;

}

#设定http服务器, which provides load balancing support using its reverse proxy function
http {
#设定mime类型
include mime.types;
Default_type Application/octet-stream;
Include/usr/local/nginx/conf/proxy.conf;
Log_format main ' $remote _addr-$remote _user [$time _local] "$request" '
' $status $body _bytes_sent "$http _referer "'
'" $http _user_agent "" $http _x_forwarded_for "';

Access_log Logs/access.log main; #设定请求缓冲server_names_hash_bucket_size 128;client_header_buffer_size 32k;large_ Client_header_buffers 4 32k;# client_max_body_size 8m;sendfile on;tcp_nopush on;tcp_nodelay on; #keepali Ve_timeout 0;keepalive_timeout 65;gzip on;gzip_min_length 1k;gzip_buffers 4 16k;gzip_http_version 1.1;gzip_comp_lev    El 2;gzip_types text/plain application/x-javascript text/css application/xml;gzip_vary on;   #此处为你tomcat的地址, you can write multiple tomcat addresses upstream Tomcat_pool {server 192.168.254.133:8080 weight=4 max_fails=2 fail_timeout=30s;   Server 192.168.254.132:8080 weight=4 max_fails=2 fail_timeout=30s;    Server 192.168.254.131:8080 weight=4 max_fails=2 fail_timeout=30s;}    server {Listen 80; server_name www.web2.com; #此处替换为你自己的网址, if there are multiple intermediate spaces index index.jsp index.htm index.html index.do; #设定访问的默认首页地址 ro Ot/home/www/web;    #设定网站的资源存放路径 #charset Koi8-r;    #access_log Logs/host.access.log Main; Location/{# root HTml    Index index.jsp index.html index.htm; } location ~ \.        (JSP|JSPX|DP)? $ #所有JSP的页面均交由tomcat处理 {proxy_set_header Host $host;         Proxy_set_header X-real-ip $remote _addr; Proxy_pass Http://tomcat_pool, #转向tomcat处理} #设定访问静态文件直接读取不经过tomcat location ~. *\.     (HTM|HTML|GIF|JPG|JPEG|PNG|BMP|SWF|IOC|RAR|ZIP|TXT|FLV|MID|DOC|PPT|PDF|XLS|MP3|WMA) $ {expires 30d; } location ~. *\.     (JS|CSS)? $ {Expires 1h;    } access_log/usr/local/nginx/logs/ubitechtest.log main; #设定访问日志的存放路径 error_page 502 503 504/50x.html;    Location =/50x.html {root html;           # Pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # #location ~ \.php$ {# root    html    # Fastcgi_pass 127.0.0.1:9000;    # Fastcgi_index index.php;    # Fastcgi_param Script_filename/scripts$fastcgi_script_name;    # include Fastcgi_params; #} # Deny access to. htaccess filES, if Apache ' s document Root # concurs with Nginx ' s one # #location ~/\.ht {# deny all;    #}}server {Listen 80;    server_name bbs.yourdomain.com; Location/{root/home/www/web/springmvc; #设定网站的资源存放路径 index index.jsp index.htm index.html index.do WELC ome.jsp; #设定访问的默认首页地址} location ~ \.        (JSP|JSPX|DP)? $ #所有JSP的页面均交由tomcat处理 {proxy_set_header Host $host;         Proxy_set_header X-real-ip $remote _addr; Proxy_pass Http://tomcat_pool, #转向tomcat处理} #设定访问静态文件直接读取不经过tomcat location ~. *\.     (HTM|HTML|GIF|JPG|JPEG|PNG|BMP|SWF|IOC|RAR|ZIP|TXT|FLV|MID|DOC|PPT|PDF|XLS|MP3|WMA) $ {expires 30d; } location ~. *\.     (JS|CSS)? $ {Expires 1h;    } access_log/usr/local/nginx/logs/ubitechztt.log main; #设定访问日志的存放路径 error_page 502 503 504/50x.html;    Location =/50x.html {root html; }}# Another virtual host using mix of ip-, name-, and port-based Configuration# #server {# Listen 8000;# listen somename:8080;# server_name somename alias another.alias;# Location/{# root html;# index index.html index.htm;#}#}

}

Create user www and user group www to use the file directory
/usr/sbin/groupadd www
/usr/sbin/useradd-g www www-s/sbin/nologin
Mkdir-p/home/www
chmod +w/home/www
Chown-r www:www/home/www

Vi/usr/local/nginx/conf/proxy.conf
Proxy_redirect off;
Proxy_set_header Host $host;
Proxy_set_header X-real-ip $remote _addr;
Client_max_body_size 10m;
Client_body_buffer_size 128k;
Proxy_connect_timeout 90;
Proxy_send_timeout 90;
Proxy_read_timeout 90;
Proxy_buffer_size 4k;
Proxy_buffers 4 32k;
Proxy_busy_buffers_size 64k;
Proxy_temp_file_write_size 64k;

/usr/local/nginx/sbin/nginx-t
Nginx:the configuration file/usr/local/nginx/conf/nginx.conf syntax is OK
Nginx:configuration file/usr/local/nginx/conf/nginx.conf Test is successful

Simple test
3 Nginx can access the agent's three Tomcat separately.
[Email protected] conf]# Curl 192.168.254.131
2222222222222222222222222222222222222222
[Email protected] conf]# Curl 192.168.254.131
33333333333333333333333333333
[Email protected] conf]# Curl 192.168.254.131
111111111111111111111111111111111111111
OK, here, Nginx and Tomcat are done.
(Note: Each of these configurations is the same)

Install lvs+keepalived.

Yum-y Install Ipvsadm keepalived (yum installation is not much to say)

Turn on route forwarding
Permanently open
echo "Net.ipv4.ip_forward = 1" >>/etc/sysctl.conf
SYSCTL-P----load, making the configuration file effective immediately
Check:
Sysctl-a |grep "Ip_forward"
Net.ipv4.ip_forward = 1
NET.IPV4.IP_FORWARD_USE_PMTU = 0

Configuring Keepalived (Master Configuration)
Vi/etc/keepalived/keepalived.conf
! Configuration File for Keepalived

Global_defs {
Notification_email {br/>[email protected]

Notification_email_from [email protected]
Smtp_server 127.0.0.1
Smtp_connect_timeout 30
router_id Lvs_master #备份服务器上将MASTER改为BACKUP
}

Vrrp_instance Vi_1 {
State BACKUP #备份服务器上将MASTER改为BACKUP
Interface Ens32 #该网卡名字需要查看具体服务器的网口
VIRTUAL_ROUTER_ID 51
Priority 100 # Change 100 to 90 on backup service
Nopreempt
Advert_int 1
Authentication {
Auth_type PASS
Auth_pass 1111
}
virtual_ipaddress {
192.168.254.88
# (if you have more than one VIP, continue with line wrapping.)
}
}

Virtual_server 192.168.254.88 80 {
Delay_loop 6 # (query Realserver status every 6 seconds)
Lb_algo RR # (RR algorithm)
Lb_kind DR # (Direct Route)
Persistence_timeout 50 # (connection of the same IP is assigned to the same realserver within 60 seconds)
Protocol TCP # (check realserver status with TCP protocol)

real_server 192.168.254.131 80 {    weight 1   #(权重)    TCP_CHECK {        connect_timeout 10    #(10秒无响应超时)        nb_get_retry 3        delay_before_retry 3        connect_port 80    } } real_server 192.168.254.132 80 {    weight 1    TCP_CHECK {        connect_timeout 10        nb_get_retry 3        delay_before_retry 3        connect_port 80    } } real_server 192.168.254.133 80 {    weight 1    TCP_CHECK {        connect_timeout 10        nb_get_retry 3        delay_before_retry 3        connect_port 80    } }

}

Backup configuration
Vi/etc/keepalived/keepalived.conf
! Configuration File for Keepalived

Global_defs {
Notification_email {br/>[email protected]

Notification_email_from [email protected]
Smtp_server 127.0.0.1
Smtp_connect_timeout 30
router_id Lvs_backup #备份服务器上将MASTER改为BACKUP
}
Vrrp_instance Vi_2 {
State BACKUP #备份服务器上将MASTER改为BACKUP
Interface Ens32 #该网卡名字需要查看具体服务器的网口
VIRTUAL_ROUTER_ID 51
Priority 90 # Change 100 to 90 on backup service
Nopreempt
Advert_int 1
Authentication {
Auth_type PASS
Auth_pass 1111
}
virtual_ipaddress {
192.168.254.88
# (if you have more than one VIP, continue with line wrapping.)
}
}

Virtual_server 192.168.254.88 80 {
Delay_loop 6 # (query Realserver status every 6 seconds)
Lb_algo RR # (RR algorithm)
Lb_kind DR # (Direct Route)
Persistence_timeout 50 # (connection of the same IP is assigned to the same realserver within 60 seconds)
Protocol TCP # (check realserver status with TCP protocol)

real_server 192.168.254.131 80 {    weight 1   #(权重)    TCP_CHECK {        connect_timeout 10    #(10秒无响应超时)        nb_get_retry 3        delay_before_retry 3        connect_port 80    } } real_server 192.168.254.132 80 {    weight 1    TCP_CHECK {        connect_timeout 10        nb_get_retry 3        delay_before_retry 3        connect_port 80    } } real_server 192.168.254.133 80 {    weight 1    TCP_CHECK {        connect_timeout 10        nb_get_retry 3        delay_before_retry 3        connect_port 80    } }

}

Service keepalived Start
ENS32: <BROADCAST,MULTICAST,UP,LOWER_UP> MTU Qdisc pfifo_fast State up group default Qlen 1000
Link/ether 00:0C:29:E3:F4:FF BRD FF:FF:FF:FF:FF:FF
inet 192.168.254.135/24 BRD 192.168.254.255 scope Global Noprefixroute ENS32
Valid_lft Forever Preferred_lft Forever
inet 192.168.254.88/32 Scope Global ENS32
Valid_lft Forever Preferred_lft Forever

显示虚拟IP 启动好了 最后就是在ngnx 服务器上配置vi /etc/init.d/realserver #!/bin/bash

sns_vip=192.168.254.88
/etc/rc.d/init.d/functions
Case "$" in
Start
Ifconfig lo:0 $SNS _vip netmask 255.255.255.255 broadcast $SNS _VIP
/sbin/route add-host $SNS _vip Dev lo:0
echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
Sysctl-p >/dev/null 2>&1
echo "Realserver Start OK"
;;
Stop
Ifconfig lo:0 Down
Route del $SNS _VIP >/dev/null 2>&1
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "0" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "0" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "Realserver stoped"
;;
*)
echo "Usage: $ {start|stop}"
Exit 1
Esac
Exit 0

chmod 755/etc/init.d/realserver
Service Realserver Start
Lo: <LOOPBACK,UP,LOWER_UP> MTU 65536 qdisc noqueue State UNKNOWN Group default Qlen 1000
Link/loopback 00:00:00:00:00:00 BRD 00:00:00:00:00:00
inet 127.0.0.1/8 Scope host Lo
Valid_lft Forever Preferred_lft Forever
inet 192.168.254.88/32 BRD 192.168.254.88 Scope Global lo:0
Valid_lft Forever Preferred_lft Forever

看到虚拟IP 就好了（同理在其他nginx服务器都要操作）测试 [[email protected] ~]# curl 192.168.254.88

444444444444444444444444
[email protected] ~]# Curl 192.168.254.88
55555555555555555555555555
[email protected] ~]# Curl 192.168.254.88
22222222222222222222222222222222222222222222222222222
[email protected] ~]# Curl 192.168.254.88
444444444444444444444444
[email protected] ~]# Curl 192.168.254.88
33333333333333333333333333333
[email protected] ~]# Curl 192.168.254.88
111111111111111111111111111111111111111

Lvs+keepalived+nginx implementation of high performance load Balancing cluster high performance JSP cluster