Robust, highly available 7-tier load Balancing scheme based on Keepalived+nginx deployment 20151214

High availability is a commonplace problem, open source high-availability software has been done quite mature, before also in Debian under the lvs+heartbeat of the 4-layer lb, has been very stable (unfortunately the flow is not very good), now because of business needs, to do a keepalived+ based on the Nginx High-Availability 7-layer load balancing.

Topology is also relatively simple, do not draw topology diagram: 2 nodes are installed on the configuration keepalived and Nginx, configure the Nginx reverse proxy backend Real server

A few key points to compare:

1, in order to avoid multiple keepalived in the same LAN in the multicast response to each other, the use of unicast communication

2. The process of state switching triggers email notifications, SMS notifications, Web notifications, log records, and facilitates understanding of the primary and standby work status through various means

3, Nginx detection script adopted a lightweight way:"killall-0 nginx", you can also use the pidof nginx way or call other custom detection scripts

4, pay special attention to the size of the priority and the change of weight when the anomaly is detected

5. Understand the working mechanism of free ARP

6, understand the scope of application of VRRP protocol: LAN, first hop gateway redundancy

7, a single VRRP instance work in the main standby mode, to maximize the use of 2 nodes of resources, can do multiple VRRP instances, to achieve high availability and load balancing

To facilitate the management of software packages, the use of CentOS keepalived,nginx1.8.0 using the official source of Nginx, the overall installation is relatively simple.

#yum Install keepalived nginx-y

Set up boot for critical services

#chkconfig Nginx on
#chkconfig keepalived on

Check the keepalived package installs those files (the documentation must look good):

#rpm-QL keepalived

/etc/keepalived
/etc/keepalived/keepalived.conf
/etc/rc.d/init.d/keepalived
/etc/sysconfig/keepalived
/usr/bin/genhash
/usr/libexec/keepalived
/usr/sbin/keepalived
/usr/share/doc/keepalived-1.2.13
/usr/share/doc/keepalived-1.2.13/author

.........

Keepalived main configuration file, 2 machine configuration file slightly different, see the configuration file in the comments

[Email protected] ~]# more/etc/keepalived/keepalived.conf
# # # #Configuration File for keepalived
# # # #内部API网关 keepalived ha Configuration
# # # Laijingli 20151213

Global_defs {
Notification_email {
[Email protected]
}
Notification_email_from [email protected]
Smtp_server 127.0.0.1
Smtp_connect_timeout 30
router_id proxy101 # proxy101 on master101, proxy102 on backup102
}

# # #simple Check with killall-0 which was less expensive than pidof to verify the Nginx is running
Vrrp_script Chk_nginx {
Script "killall-0 Nginx"
Interval 1
Weight 2
Fall 2
Rise 1
}

Vrrp_instance Yn_api_gateway {
State MASTER # # MASTER on master101, BACKUP on backup102
Interface Em1
VIRTUAL_ROUTER_ID 101
Priority # # # MASTER101, 199 on backup102
Advert_int 1
# # #采用单播通信 To avoid interaction between multiple keepalived groups in the same LAN
Unicast_src_ip 192.168.0.101 # #本机ip
Unicast_peer {
192.168.0.102 # #对端ip
}
Authentication {
Auth_type PASS
Auth_pass Testpass
}
virtual_ipaddress {
192.168.0.105 # # VIP
}
# # #如果只有一块网卡的话监控网络接口就没有必要了
#track_interface {
#Em1
#}
Track_script {
Chk_nginx
}
# # #状态切换是发送邮件通知, this machine record log, late will trigger SMS notification
Notify_master/usr/local/bin/keepalived_notify.sh Notify_master
Notify_backup/usr/local/bin/keepalived_notify.sh Notify_backup
Notify_fault/usr/local/bin/keepalived_notify.sh Notify_fault
notify/usr/local/bin/keepalived_notify.sh Notify
Smtp_alert
}


VRRP Instance Group node state toggle trigger mail notification, SMS notification, log record script
# more/usr/local/bin/keepalived_notify.sh
#!/bin/bash
# # #keepalived notify script for record ha state transtion to log files

# # #将将状态转换过程记录到log, easy to debug
Logfile=/var/log/keepalived.notify.log
Echo--------------->> $logfile
echo ' Date ' [' hostname '] keepalived HA role State transition: $ $ $4 $ $6 >> $logfile

# # #将状态转换记录到nginx的文件, easy to view ha status through the Web (be careful not to open to the public network)
echo ' Date ' hostname ' $ $4 $ $6 >/usr/share/nginx/html/index.html

Nginx part of the configuration file, for reference only

# more/etc/nginx/nginx.conf

# # #运维管理用途: Used to differentiate VIP running on that server
server {
Listen 80;
server_name localhost;

Location/{
root/usr/share/nginx/html;
Index index.html index.htm;
}

# # Nginx Monitor Use only
# # #add by Lai monitor nginx status
Location/server-status {
Stub_status on;
Allow 127.0.0.1;
Allow 192.168.0.0/24;
}
}

Simulate the failure by starting and stopping keepalived and nginx to test the state switching process of the VRRP instance (that is, the VIP drift):

/etc/init.d/keepalived Start/stop

/etc/init.d/nginx Stop start/stop

To view the VIP running on that server:

# IP Addr show|grep 192.168
inet 192.168.0.101/24 BRD 192.168.0.255 Scope Global EM1
inet 192.168.0.105/32 Scope Global EM1

Test:

[Email protected] ~]$ Curl 192.168.0.101
Mon Dec 16:27:10 CST proxy101 INSTANCE yn_api_gateway MASTER 202
[Email protected] ~]$ Curl 192.168.0.102
Mon Dec 16:34:40 CST proxy102 INSTANCE yn_api_gateway BACKUP 199
[Email protected] ~]$ Curl 192.168.0.105
Mon Dec 16:27:10 CST proxy101 INSTANCE yn_api_gateway MASTER 202

# Tail/var/log/keepalived.notify.log
Mon Dec 16:25:13 CST [proxy101] keepalived HA role State transition:
Mon Dec 16:25:13 CST [proxy101] keepalived HA role State transition:instance Yn_api_gateway MASTER 202
---------------
---------------
Mon Dec 16:26:34 CST [proxy101] keepalived HA role State transition:
Mon Dec 16:26:34 CST [proxy101] keepalived HA role State transition:instance Yn_api_gateway BACKUP 200
---------------
---------------
Mon Dec 16:27:10 CST [proxy101] keepalived HA role State transition:
Mon Dec 16:27:10 CST [proxy101] keepalived HA role State transition:instance Yn_api_gateway MASTER 202

# tail/var/log/messages
Dec 16:27:08 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) forcing a new MASTER election
Dec 16:27:08 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) forcing a new MASTER election
Dec 16:27:09 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) Transition to MASTER State
Dec 16:27:10 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) Entering MASTER State
Dec 16:27:10 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) setting protocol VIPs.
Dec 16:27:10 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) sending gratuitous ARPs on EM1 for 192.168 .0.105
Dec 16:27:10 localhost keepalived_healthcheckers[74307]: NetLink Reflector reports IP 192.168.0.105 added
Dec 16:27:10 localhost keepalived_vrrp[74308]: Remote SMTP Server [127.0.0.1]:25 connected.
Dec 16:27:10 localhost keepalived_vrrp[74308]: SMTP alert successfully sent.
Dec 16:27:15 localhost keepalived_vrrp[74308]: vrrp_instance (yn_api_gateway) sending gratuitous ARPs on EM1 for 192.168 .0.105

Primary and Standby Status transition notification message:

Find out more about the work process by grasping the package view:

# Tcpdump-ni Em1 VRRP
Tcpdump:verbose output suppressed, use-v OR-VV for full protocol decode
Listening on em1, Link-type EN10MB (Ethernet), capture size 65535 bytes
17:36:47.098225 IP 192.168.0.101 > 192.168.0.102:vrrpv2, advertisement, Vrid 101, Prio 202, authtype simple, intvl 1s, Length 20
17:36:47.388540 IP 192.168.0.22 > 224.0.0.18:vrrpv2, advertisement, Vrid Wuyi, Prio, authtype simple, intvl 1s, leng Th 20
17:36:48.099409 IP 192.168.0.101 > 192.168.0.102:vrrpv2, advertisement, Vrid 101, Prio 202, authtype simple, intvl 1s, Length 20
17:36:48.389504 IP 192.168.0.22 > 224.0.0.18:vrrpv2, advertisement, Vrid Wuyi, Prio, authtype simple, intvl 1s, leng Th 20
17:36:49.100544 IP 192.168.0.101 > 192.168.0.102:vrrpv2, advertisement, Vrid 101, Prio 202, authtype simple, intvl 1s, Length 20
17:36:49.390487 IP 192.168.0.22 > 224.0.0.18:vrrpv2, advertisement, Vrid Wuyi, Prio, authtype simple, intvl 1s, leng Th 20
17:36:50.101713 IP 192.168.0.101 > 192.168.0.102:vrrpv2, advertisement, Vrid 101, Prio 202, authtype simple, intvl 1s, Length 20
17:36:50.391453 IP 192.168.0.22 > 224.0.0.18:vrrpv2, advertisement, Vrid Wuyi, Prio, authtype simple, intvl 1s, leng Th 20

Reference: https://www.nginx.com/resources/admin-guide/nginx-ha-keepalived/

Robust, highly available 7-tier load Balancing scheme based on Keepalived+nginx deployment 20151214