Build highly available reverse proxies with Haproxy and keepalived

Build highly available reverse proxies with Haproxy and keepalived

Building high-availability reverse proxies with haproxy and keepalived in the case of a site with a large number of accesses, as the traffic increases, a single server is unable to process all the requests, which requires multiple servers to divert a large amount of requests, that is, load balancing. In the case of load balancing, these requests must be distributed on the portal's Portal deployment server (not just one), which is the reverse proxy. Because the reverse proxy server is the portal of the site, its load pressure and vulnerable, there is a single point of failure risk, so we need a highly available scenario to achieve when a reverse proxy server down, another server will automatically take over the service. Based on the above requirements, we use haproxy,keepalived to build highly available reverse proxy systems. Introduction Haproxy is a high-performance proxy server, which can provide 7-tier and 4-tier agents, with Healthcheck, load balancing and other features, performance excellence, many large internet companies in the country are in use, keepalived is a highly available program, through the VIP (that is, virtual IP) and heartbeat detection to achieve high availability. The principle is that there is a set of (two) servers, respectively, to give master,backup two roles, by default, Master will bind the VIP to their own network card, to provide services. Master,backup will send a heartbeat packet to each other at a certain time interval to detect the other party's status, the interval is typically 2 seconds, if backup finds master down, then backup will send ARP packet to the gateway, the VIP bound to their network card, At this point, Backup provides services for automated failover, which will re-take over the service when master resumes.

Deployment Environment Readiness

Environment Os:centos Linux Release 6.0 (Final) 2.6.32-71.29.1.el6.x86_64

haproxy:1.5.10

keepalived:1.2.2

vip:192.168.36.100

m:192.168.36.154

s:192.168.36.129

Architecture

                   192.168.36.100             +-----------VIP----------+                |                        |             |                        |           Master                   Backup        192.168.36.129            192.168.36.154        +----------+             +----------+        | HAProxy  |             | HAProxy  |        |keepalived|             |keepalived|        +----------+             +----------+             |               v      +--------+---------+     |        |         |    |        |         |    v        v         v+------+  +------+  +------+| WEB1 |  | WEB2 |  | WEB3 |+------+  +------+  +------+

Installing Haproxy

Installing Pcre

 yum install pcre wget wget http://www.haproxy.org/download/1.5/src/haproxy-1.5.10.tar.gz tar -zxvf haproxy-1.5.10.tar.gz cd haproxy-1.5.10注意编译参数： TARGET是指自己系统的内核版本 ARCH指定系统是32位还是64位 CPU=native: use the build machine‘s specific processor optimizations 更多编译参数内容见源码中的README  make TARGET=linux26 make TARGET=linux26 USE_OPENSSL=1 ADDLIB=-lz  #编译支持ssl认证功能，haproxy 1.5t版本以上才支持。 make install ln -s /usr/local/src/haproxy-1.5.10 /usr/local/haproxy mkdir /usr/local/haproxy/etc

Configuration file Hparoxy

CD haproxy-1.5.10cat etc/haproxy.cfgglobal log 127.0.0.1 local3 info maxconn 20000 User Lashourun Group               Lashourun chroot/var/chroot/haproxy daemon nbproc 4defaults log 127.0.0.1 local3 mode http Option httplog option httpclose option dontlognull option forwardfor option Accept-invalid-http-req Uest Retries 2 Balance roundrobin stats enable stats uri/admin stats Auth haadmin:lashoupay timeout HT Tp-request 20s Timeout Connect 50000 timeout client 500000 timeout server 500000frontend lashoupay-http #支持http 80 port Access to the business bind *:80 reqadd x-forwarded-proto:\ http #acl lashoupay-4.90 DST 192.168.4.90 ACL lashoupay-4.90 path_b Eg/lashoupay/lashoupay | | DST 192.168.4.90 ACL vs-lashoupay-index-4.94 DST 192.168.4.111 #acl vs-lsmvas-4.51 DST 192.168.4.51 #acl vs-lsmva s-4.93 DST 192.168.4.93 ACL vs-pay-4.90 DST 192.168.4.114 ACL vs-project-4.95 DST 192.168.4.115 use_backend LAshoupay-4.90-pool if lashoupay-4.90 use_backend vs-lashoupay-index-4.94-pool if vs-lashoupay-index-4.94 #use_backen D vs-lsmvas-4.51-pool if vs-lsmvas-4.51 #use_backend vs-lsmvas-4.93-pool if vs-lsmvas-4.93 use_backend vs-pay-4.90-p    ool if vs-pay-4.90 use_backend vs-project-4.95-pool if vs-project-4.95frontend Lashoupay-https #支持https 443 SSL certified access to Business Bind *:443 SSL crt/usr/local/haproxy/cert.pem option httpclose option forwardfor reqadd x-forwarded-proto:\ h TTPs default_backend lashoupay-4.90-poolfrontend vs-lsmvas4.51-9002 #支持一些特殊业务端口访问 mode http bind *:9002 Reqad D x-forwarded-proto:\ http ACL vs-lsmvas-4.51 DST 192.168.4.51 ACL vs-lsmvas-4.93 DST 192.168.4.93 use_backend vs -lsmvas-4.51-pool if vs-lsmvas-4.51 use_backend vs-lsmvas-4.93-pool if Vs-lsmvas-4.93backend lashoupay-4.90-pool red Irect scheme HTTPS if! {SSL_FC} balance Roundrobin stats refresh 2 server lashoupay-4.73 192.168.4.73:80 weight 3 Maxconn 1000 0 Checkbackend vs-lashoupay-index-4.94-pool balance Roundrobin stats Refresh 2 server lashoupay-index-16-26 192.16 8.4.16:9002 Weight 3 maxconn 10000 checkbackend vs-lsmvas-4.51-pool balance Roundrobin stats Refresh 2 #server vs -lsmvas-4.16 192.168.4.16:9002 weight 3 maxconn 10000 Check server vs-lsmvas-4.18 192.168.4.18:9002 weight 3 maxconn 10 Check server vs-lsmvas-4.26 192.168.4.26:9002 weight 3 maxconn 10000 Check server vs-lsmvas-4.28 192.168.4.28:90 Weight 3 maxconn 10000 checkbackend vs-lsmvas-4.93-pool balance Roundrobin stats Refresh 2 server vs-lsmvas-4. 192.168.4.18:9002 Weight 3 maxconn 10000 Check server vs-lsmvas-4.28 192.168.4.28:9002 weight 3 maxconn 10000 CHECKB Ackend vs-pay-4.90-pool balance Roundrobin stats Refresh 2 #server vs-lsmvas-4.16 192.168.4.16:9002 weight 3 maxc Onn 10000 Check server vs-lsmvas-4.18 192.168.4.18:9002 weight 3 maxconn 10000 Check server vs-lsmvas-4.26 192.168.4 .26:9002 Weight 3 Maxconn 10000 Check server vs-lsmvas-4.28 192.168.4.28:9002 weight 3 maxconn 10000 Checkbackend Vs-project-4.95-pool Balan Ce Roundrobin stats Refresh 2 server project-4.17 192.168.4.17:9002 weight 3 maxconn 10000 Check server project-4 .192.168.4.27:9002 Weight 3 maxconn 10000 Check

To view the status of Haproxy: Http://192.168.36.100/haproxy-stats, this page shows the status of the haproxy itself and the backend server.

Log haproxy will send the log records to the Syslog server (CentOS6 is rsyslogd,udp514 port), edit the/etc/rsyslog.conf file, add the following:

ModLoad imudpUDPServerRun 514UDPServerAddress 127.0.0.1local3.*                /var/log/haproxy.log

Restart Rsyslog

 /etc/init.d/rsyslog restart

Automatically rotate logs, edit/etc/logrotate.d/haproxy.cfg, add the following:

/var/log/haproxy.log{    rotate 4    daily    missingok    notifempty    compress    delaycompress    sharedscripts    postrotate    reload rsyslog > /dev/null 2>&1 || true    endscript}

Startup scripts

 wget -O haproxy https://raw.github.com/gist/3665034/4125bd5b81977a72e5eec30650fb21f3034782a0/haproxy-init.d  cp haproxy /etc/init.d/haproxy chmod +x /etc/init.d/haproxy#使用方式 /etc/init.d/haproxy start|stop|restart

Installing keepalived

Installing dependent libraries

 yum install popt popt-devel wget http://www.keepalived.org/software/keepalived-1.2.2.tar.gz tar -zxvf keepalived-1.2.2.tar.gz cd keepalived-1.2.2 ./configure --prefix=/usr/local/keepalived make && make install cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/keepalived cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ mkdir -p /etc/keepalived/ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf  chmod +x /etc/init.d/keepalived

How to use

 /etc/init.d/keepalived start|stop|restart

The configuration on the master server

cat etc/keepalived/keepalived.confglobal_defs {   notification_email {       [email protected]   }   notification_email_from [email protected]   smtp_server 192.168.x.x   smtp_connect_timeout 30   router_id LVS_DEVEL}#监测haproxy进程状态，每2秒执行一次vrrp_script chk_haproxy {    script "/usr/local/keepalived/chk_haproxy.sh"    interval 2    weight 2}vrrp_instance VI_1 {    state MASTER #标示状态为MASTER    interface eth0    virtual_router_id 51    priority 101   #MASTER权重要高于BACKUP    advert_int 1    mcast_src_ip 192.168.36.129 #Master服务器IP    authentication {        auth_type PASS #主从服务器验证方式        auth_pass 1111    }    track_script {        chk_haproxy #监测haproxy进程状态    }    #VIP    virtual_ipaddress {        192.168.36.100 #虚拟IP    }}

Configuration on the Bakcup server

cat etc/keepalived/keepalived.confglobal_defs {   notification_email {   [email protected]   }   notification_email_from [email protected]   smtp_server 192.168.x.x   smtp_connect_timeout 30   router_id LVS_DEVEL}#监测haproxy进程状态，每2秒执行一次vrrp_script chk_haproxy {    script "/usr/local/keepalived/chk_haproxy.sh"    interval 2    weight 2}vrrp_instance VI_1 {    state BACKUP #状态为BACKUP    interface eth0    virtual_router_id 51    priority 100  #权重要低于MASTER    advert_int 1    mcast_src_ip 192.168.36.154 #Backup服务器的IP    authentication {        auth_type PASS        auth_pass 1111    }    track_script {        chk_haproxy #监测haproxy进程状态    }    #VIP    virtual_ipaddress {        192.168.36.100 #虚拟IP    }}

• Script chk_haproxy.sh content referenced in keepalived configuration

#!/bin/bash## author: weizhifeng# description: # 定时查看haproxy是否存在，如果不存在则启动haproxy，# 如果启动失败，则停止keepalived# status=(ps aux|grep haproxy | grep -v grep | grep -v bash | wc -l)if [ "{status}" = "0" ]; then    /etc/init.d/haproxy start    status2=(ps aux|grep haproxy | grep -v grep | grep -v bash |wc -l)    if [ "{status2}" = "0"  ]; then            /etc/init.d/keepalived stop    fifi

Highly Available tests

• 1. Stop keepalived on master, view system logs, send IP

  /etc/init.d/keepalived stop  tail -f /var/log/message Keepalived: Terminating on signal Keepalived: Stopping Keepalived v1.2.2 (11/03,2011)  Keepalived_vrrp: Terminating VRRP child process on signal  Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

• 2. Check the system log on backup and find that backup has entered the master role and that the VIP 192.168.36.100 is bound

  tail -f /var/log/message Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.36.100 #在Backup上查看VIP是否已经绑定

• 3. Restart keepalived on master, view the system log, find the master role regain, and bind VIP 192.168.36.100

  /etc/init.d/keepalived start  tail -f /var/log/message Keepalived_vrrp: VRRP_Instance(VI_1) Transition to MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) Entering MASTER STATE Keepalived_vrrp: VRRP_Instance(VI_1) setting protocol VIPs. Keepalived_vrrp: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.36.100

• 4. Review the System log on backup, find it back to the backup role, and release the VIP

  tail -f /var/log/message Keepalived_vrrp: VRRP_Instance(VI_1) Received higher prio advert Keepalived_vrrp: VRRP_Instance(VI_1) Entering BACKUP STATE Keepalived_vrrp: VRRP_Instance(VI_1) removing protocol VIPs.

• Concurrency test we use webbench to perform concurrency tests on haproxy

 yum install ctags wget http://home.tiscali.cz/~cz210552/distfiles/webbench-1.5.tar.gz tar -zxvf webbench-1.5.tar.gz cd webbench-1.5 make  mkdir -p /usr/local/man && make install

Test method:

 webbench -c 100 -t 3000 http://192.168.36.100/index.html

Haproxy Maintenance

• 1,haproxy do front-end agent, error class -1/-1/-1/-1/0 187--pr--0/0/0/0/0 0/0 "

  Google a bit is full of English, probably the reason is because the client sent an invalid character, can not handle, plainly, is probably not support the customer sent the character set,

  We looked up the string sent by the Chinese, as if it is gb2312, Google has the cattle people to give the solution is as follows:

  Open the HAPROXY.CFG configuration plus the following

  Timeout Http-request 20s

  Option Accept-invalid-http-request

Build highly available reverse proxies with Haproxy and keepalived