Deploy a robust high-availability layer-7 Load Balancing solution based on keepalived + nginx 20151214

Deploy a robust high-availability layer-7 Load Balancing solution based on keepalived + nginx 20151214

High Availability is a common topic. Open-source high availability software has been developed quite well. Previously, it has done lvs + heartbeat 4-layer LB In debian, it has been very stable (unfortunately, the traffic is not large). Now, due to business needs, we have implemented a keepalived + nginx-based high-availability layer-7 load balancing.

 

The topology structure is also relatively simple, so no topology is shown: install and configure keepalived and nginx on the two nodes respectively, and configure the nginx reverse proxy backend real server

 

Key points:

1. To prevent multicast responses from multiple keepalived groups in the same LAN, unicast communication is adopted.

2. Trigger mail Notification, SMS notification, web notification, and log record during status switching, so that you can understand the active/standby status through various channels.

3. The nginx detection script is lightweight: "killall-0 nginx". You can also use pidof nginx or call other custom detection scripts.

4. Pay special attention to the priority and the weight changes when exceptions are detected.

5. Understand the working mechanism of free ARP

6. Understand the applicability of VRRP Protocol: Lan, first-hop gateway Redundancy

7. A single vrrp instance works in master-slave mode. To maximize the utilization of resources on two nodes, multiple vrrp instances can be used to achieve high availability and load balancing.

 

 

To facilitate software package management, use keepalived provided by centos, nginx1.8.0 adopts nginx official source, and the overall installation is relatively simple.

# Yum install keepalived nginx-y

 

Set the startup of key services

# Chkconfig nginx on
# Chkconfig keepalived on

 

Check the files installed in the keepalived package (the document must be nice ):

# Rpm-ql keepalived

/Etc/keepalived
/Etc/keepalived. conf
/Etc/rc. d/init. d/keepalived
/Etc/sysconfig/keepalived
/Usr/bin/genhash
/Usr/libexec/keepalived
/Usr/sbin/keepalived
/Usr/share/doc/keepalived-1.2.13
/Usr/share/doc/keepalived-1.2.13/AUTHOR

.........

 

 

The master configuration file of keepalived. The configuration files of the two machines are slightly different. For details, see the notes in the configuration file.

[Root @ proxy101 ~] # More/etc/keepalived. conf
#### Configuration File for keepalived
#### Keepalived HA configuration of the internal API Gateway
#### Laijingli20151213

Global_defs {
Notification_email {
[Email protected]
}
Notification_email_from [email protected]
Smtp_server 127.0.0.1
Smtp_connect_timeout 30
Router_id proxy101 # proxy101 on master101, proxy102 on backup102
}

### Simple check with killall-0 which is less expensive than pidof to verify that nginx is running
Vrrp_script chk_nginx {
Script "killall-0 nginx"
Interval 1
Weight 2
Fall 2
Rise 1
}

Vrrp_instance YN_API_GATEWAY {
State MASTER # MASTER on master101, BACKUP on backup102
Interface em1
Virtual_router_id 101
Priority 200 #200 on master101, 199 on backup102
Advert_int 1
### Use unicast communication to avoid mutual impact between multiple keepalived groups in the same LAN
Unicast_src_ip 192.168.0.101 # local ip Address
Unicast_peer {
192.168.0.102 # peer ip Address
}
Authentication {
Auth_type PASS
Auth_pass testpass
}
Virtual_ipaddress {
192.168.0.105 # VIP
}
### Network interface monitoring is unnecessary if only one network card is available
# Track_interface {
# Em1
#}
Track_script {
Chk_nginx
}
### Status switching refers to sending email notifications. logs are recorded on the local machine and SMS notifications are triggered later.
Notify_master/usr/local/bin/keepalived_policy.sh notify_master
Notify_backup/usr/local/bin/keepalived_policy.sh policy_backup
Notify_fault/usr/local/bin/keepalived_policy.sh policy_fault
Notify/usr/local/bin/keepalived_policy.sh restart y
Smtp_alert
}


Script for changing the status of VRRP instance group nodes to trigger mail notifications, SMS notifications, and log records
# More/usr/local/bin/keepalived_policy.sh
#! /Bin/bash
### Keepalived running y script for record ha state transtion to log files

### Record the state transition process to log to facilitate troubleshooting
Logfile =/var/log/keepalived. Logs Y. log
Echo ---------------> $ logfile
Echo 'date' ['hostname'] keepalived HA role state transition: $1 $2 $3 $4 $5 $6 >>> logfile

### Record the status transition to the nginx file for viewing the ha status through the web (Be sure not to open it to the public network)
Echo 'date' hostname' $1 $2 $3 $4 $5 $6>/usr/share/nginx/html/index.html

 

Some nginx configuration files are for reference only.

# More/etc/nginx. conf

### O & M management purpose: used to differentiate the virtual IP (vip) running on that server
Server {
Listen 80;
Server_name localhost;

Location /{
Root/usr/share/nginx/html;
Index index.html index.htm;
}

# Nginx monitor use only
### Add by lai monitor nginx status
Location/server-status {
Stub_status on;
Allow 127.0.0.1;
Allow 192.168.0.0/24;
}
}

 

 

Start and Stop keepalived and nginx to simulate faults and test the status switching process (VIP drift) of vrrp instances ):

/Etc/init. d/keepalived start/stop

/Etc/init. d/nginx stop start/stop

 

Check which server the VIP is running on:

# Ip addr show | grep 192.168
Inet 192.168.0.101/24 brd 192.168.0.255 scope global em1
Inet 192.168.0.105/32 scope global em1

 

Test:

[Root @ test28 ~] $ Curl 192.168.0.101
Mon Dec 14 16:27:10 CST 2015 proxy101 INSTANCE YN_API_GATEWAY MASTER 202
[Root @ test28 ~] $ Curl 192.168.0.102
Mon Dec 14 16:34:40 CST 2015 proxy102 INSTANCE YN_API_GATEWAY BACKUP 199
[Root @ test28 ~] $ Curl 192.168.0.105
Mon Dec 14 16:27:10 CST 2015 proxy101 INSTANCE YN_API_GATEWAY MASTER 202

 

# Tail/var/log/keepalived. Sort Y. log
Mon Dec 14 16:25:13 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:25:13 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY MASTER 202
---------------
---------------
Mon Dec 14 16:26:34 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:26:34 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY BACKUP 200
---------------
---------------
Mon Dec 14 16:27:10 CST 2015 [proxy101] keepalived HA role state transition:
Mon Dec 14 16:27:10 CST 2015 [proxy101] keepalived HA role state transition: INSTANCE YN_API_GATEWAY MASTER 202

 

# Tail/var/log/messages
Dec 14 16:27:08 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) forcing a new MASTER election
Dec 14 16:27:08 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) forcing a new MASTER election
Dec 14 16:27:09 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) Transition to MASTER STATE
Dec 14 16:27:10 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) Entering MASTER STATE
Dec 14 16:27:10 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) setting protocol VIPs.
Dec 14 16:27:10 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) Sending gratuitous ARPs on em1 for 192.168.0.105
Dec 14 16:27:10 localhost Keepalived_healthcheckers [74307]: Netlink reflector reports IP 192.168.0.105 added
Dec 14 16:27:10 localhost Keepalived_vrrp [74308]: Remote SMTP server [127.0.0.1]: 25 connected.
Dec 14 16:27:10 localhost Keepalived_vrrp [74308]: SMTP alert successfully sent.
Dec 14 16:27:15 localhost Keepalived_vrrp [74308]: VRRP_Instance (YN_API_GATEWAY) Sending gratuitous ARPs on em1 for 192.168.0.105

 

Notification Email for master-slave status transition:

 

You can view the detailed working process through packet capture:

# Tcpdump-ni em1 vrrp
Tcpdump: verbose output suppressed, use-v or-vv for full protocol decode
Listening on em1, link-type EN10MB (Ethernet), capture size 65535 bytes
17:36:47. 098225 IP 192.168.0.101> 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1 s, length 20
17:36:47. 388540 IP 192.168.0.22> 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1 s, length 20
17:36:48. 099409 IP 192.168.0.101> 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1 s, length 20
17:36:48. 389504 IP 192.168.0.22> 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1 s, length 20
17:36:49. 100544 IP 192.168.0.101> 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1 s, length 20
17:36:49. 390487 IP 192.168.0.22> 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1 s, length 20
17:36:50. 101713 IP 192.168.0.101> 192.168.0.102: VRRPv2, Advertisement, vrid 101, prio 202, authtype simple, intvl 1 s, length 20
17:36:50. 391453 IP 192.168.0.22> 224.0.0.18: VRRPv2, Advertisement, vrid 51, prio 120, authtype simple, intvl 1 s, length 20