Implement Web server load Balancing in Linux (haproxy+keepalived)

Description

Operating system: CentOS 5.X 64-bit

Web server: 192.168.21.127, 192.168.21.128

Sites: Bbs.111cn.net and Sns.111cn.net deployed on two Web servers

To achieve the purpose:

Add two servers (main main mode) to achieve Web server load balancing through haproxy+keepalived

Architecture Planning:

Haproxy server: 192.168.21.129, 192.168.21.130

Virtual Server (VIP): 192.168.21.253, 192.168.21.254

After deployment is complete:

1, vip:192.168.21.253 Point 192.168.21.129;vip:192.168.21.254 point to 192.168.21.130;

2, when 192.168.21.129 downtime, vip:192.168.21.253 drift to 192.168.21.130;

3, when 192.168.21.130 downtime, vip:192.168.21.254 drift to 192.168.21.129;

The main advantage of this is that the two servers provide services while they are each other's backup servers.

Specific actions:

Part one: Operate on two HAPROXY servers separately

First, close the SELinux, configure the firewall

1, Vi/etc/selinux/config

#SELINUX =enforcing #注释掉

#SELINUXTYPE =targeted #注释掉

Selinux=disabled #增加

: wq! #保存退出

Setenforce 0 #使配置立即生效

2, Vi/etc/sysconfig/iptables #编辑

-A rh-firewall-1-input-d 224.0.0.18-j ACCEPT #允许组播地址通信

-A rh-firewall-1-input-p vrrp-j ACCEPT #允许VRRP (Virtual Router Redundancy association) communication

-A rh-firewall-1-input-m state--state new-m tcp-p TCP--dport 80-j ACCEPT #允许80端口通过防火墙

: wq! #保存退出

/etc/init.d/iptables Restart #重启防火墙使配置生效

Second, install Haproxy

1. Create Haproxy run accounts and groups

Groupadd Haproxy #添加haproxy组

Useradd-g haproxy haproxy-s/bin/false #创建nginx运行账户haproxy并加入到haproxy组, do not allow haproxy users to log on directly to the system

2, install the compilation tool

Yum install gcc gcc-c++ make Openssl-devel kernel-devel

3. Installation Haproxy

Haproxy Download Address: http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.24.tar.gz

Upload haproxy-1.4.24.tar.gz to/usr/local/src directory

CD/USR/LOCAL/SRC #进入软件包存放目录

Tar zxvf haproxy-1.4.24.tar.gz #解压

CD haproxy-1.4.24 #进入安装目录

Make target=linux26 cpu=x86_64 prefix=/usr/local/haprpxy #编译

Make install Prefix=/usr/local/haproxy #安装

Parameter description:

Target=linux26 #使用uname-R to view the kernel, such as: 2.6.18-371.el5, this argument is linux26

Cpu=x86_64 #使用uname-R to view system information, such as x86_64 x86_64 x86_64 Gnu/linux, at which point this parameter is x86_64

Prefix=/usr/local/haprpxy #/usr/local/haprpxy for HAPRPXY installation path

4, set Haproxy

Mkdir-p/usr/local/haproxy/conf #创建配置文件目录

Mkdir-p/etc/haproxy #创建配置文件目录

Cp/usr/local/src/haproxy-1.4.24/examples/haproxy.cfg/usr/local/haproxy/conf/haproxy.cfg #拷贝配置模板文件

Ln-s/usr/local/haproxy/conf/haproxy.cfg/etc/haproxy/haproxy.cfg #添加配置文件软连接

Cp-r/usr/local/src/haproxy-1.4.24/examples/errorfiles/usr/local/haproxy/errorfiles #拷贝错误页面

Ln-s/usr/local/haproxy/errorfiles/etc/haproxy/errorfiles #添加软连接

Mkdir-p/usr/local/haproxy/log #创建日志文件目录

Touch/usr/local/haproxy/log/haproxy.log #创建日志文件

Ln-s/usr/local/haproxy/log/haproxy.log/var/log/haproxy.log #添加软连接

Cp/usr/local/src/haproxy-1.4.24/examples/haproxy.init/etc/rc.d/init.d/haproxy #拷贝开机启动文件

chmod +x/etc/rc.d/init.d/haproxy #添加脚本执行权限

Chkconfig Haproxy on #设置开机启动

Ln-s/usr/local/haproxy/sbin/haproxy/usr/sbin #添加软连接

5. Configure HAPROXY.CFG Parameters

Cp/usr/local/haproxy/conf/haproxy.cfg/usr/local/haproxy/conf/haproxy.cfg-bak #备份

Vi/usr/local/haproxy/conf/haproxy.cfg #编辑, modify

#####################################################################

# This config needs haproxy-1.1.28 or haproxy-1.2.1

Global

Log 127.0.0.1 local0 #在本机记录日志

Log 127.0.0.1 Local1 Notice

#log Loghost local0 Info

Maxconn 65535 #每个进程可用的最大连接数

Chroot/usr/local/haproxy #haproxy安装目录

UID #运行haproxy的用户uid ( cat/etc/passwd view)

GID #运行haproxy的组uid ( cat/etc/group view)

Daemon #以后台守护进程运行

Pidfile/usr/local/haproxy/haproxy.pid #将所有进程写入pid文件

#debug #调试模式

#quiet #安装模式

Defaults

#log Global

Log 127.0.0.1 Local3 #日志文件设置

Mode HTTP #运行模式tcp, HTTP, health

Option Httplog

Option Httpclose #每次请求完毕后主动关闭http通道

Option Dontlognull #不记录健康检查的日志信息

Option Redispatch #如果后端有服务器宕机, forcing a switch to a normal server

Option Abortonclose #丢弃由于客户端等待时间过长而关闭连接但仍在haproxy等待队列中的请求

Option Forwardfor #获取客户端真实访问IP

Option Originalto #记录客户端访问的目的IP

Maxconn 65535 #每个进程可用的最大连接数

Balance Source #同一IP地址的所有请求都发送到同一服务器

Retries 3 #三次连接失败, the service is not available

Contimeout 5000 #连接超时

Clitimeout 50000 #客户端超时

Srvtimeout 50000 #服务器超时

Timeout Check 5s #检测超时

Stats Refresh 30s #统计页面自动刷新时间

Stats Uri/haproxy-status #统计页面URL路径

Stats Realm Haproxy-status #统计页面输入密码框提示信息

Stats Auth admin:123456 #统计页面用户名和密码

Stats Hide-version #隐藏统计页面上HAProxy版本信息

Frontend Web #自定义描述信息

Bind:80 #监听80端口

ACL bbs.111cn.net HDR (host)-I bbs.111cn.net #规则设置, I followed by the domain name to be accessed, if access to the bbs.111cn.net domain, load balanced to bbs.111cn.net scope

Use_backend bbs.111cn.net if Bbs.111cn.net #acl和if后面的名称必须相同这里为bbs. 111cn.net

ACL sns.111cn.net HDR (host)-I sns.111cn.net #规则设置, I followed by the domain name to be accessed, if access to the sns.111cn.net domain, load balanced to sns.111cn.net scope

Use_backend sns.111cn.net If sns.111cn.net

Backend Bbs.111cn.net

Mode http

Balance Source

#option httpchk/index.php #检测服务器此文件是否存在, if not, the server connection exception is considered, this parameter can not set

Server 192.168.21.127 192.168.21.127:80 check Inter rise 3 Fall 3 weight #inter 2000 heartbeat detection time; rise 33 connection succeeded, the server is normal, Fall 33 times Connection failed, indicates server exception, weight 100 weight setting

Server 192.168.21.128 192.168.21.128:80 check Inter rise 3 Fall 3 Weight 100

Backend Sns.111cn.net

Mode http

Balance source #设置负载均衡模式, source saves the session value, Roundrobin polling mode

#option httpchk/index.php #检测服务器此文件是否存在, if not, the server connection exception is considered, this parameter can not set

Server 192.168.21.127 192.168.21.127:80 check Inter rise 3 Fall 3 Weight 100

Server 192.168.21.128 192.168.21.128:80 check Inter rise 3 Fall 3 Weight 100

#errorloc 503 http://www.111cn.net/404.html

ErrorFile 403/etc/haproxy/errorfiles/403.http

ErrorFile 500/etc/haproxy/errorfiles/500.http

ErrorFile 502/etc/haproxy/errorfiles/502.http

ErrorFile 503/etc/haproxy/errorfiles/503.http

ErrorFile 504/etc/haproxy/errorfiles/504.http

#####################################################################

: wq! #保存退出

Service Haproxy Start #启动

Service Haproxy Stop #关闭

Service Haproxy Restart #重启

6. Set up Haproxy log

Vi/etc/syslog.conf #编辑, increase at the bottom

# Haproxy.log

local0.*/var/log/haproxy.log

local3.*/var/log/haproxy.log

: wq! #保存退出

Vi/etc/sysconfig/syslog #编辑修改

Syslogd_options= "-r-m 0" #接收远程服务器日志

: wq! #保存退出

Service syslog Restart #重启syslog

Iii. installation of keepalived

Download keeplived:http://www.keepalived.org/software/keepalived-1.2.12.tar.gz

Upload keepalived-1.2.12.tar.gz to/usr/local/src directory

Cd/usr/local/src

Tar zxvf keepalived-1.2.12.tar.gz

CD keepalived-1.2.12

./configure #配置, you must see the following prompts that the configuration is correct to continue the installation

Use Ipvs Framework:yes

Ipvs Sync Daemon Support:yes

Use VRRP Framework:yes

Make #编辑

Make install #安装

cp/usr/local/etc/sysconfig/keepalived/etc/sysconfig/

Mkdir/etc/keepalived

cp/usr/local/etc/keepalived/keepalived.conf/etc/keepalived/

cp/usr/local/sbin/keepalived/usr/sbin/

cp/usr/local/etc/rc.d/init.d/keepalived/etc/rc.d/init.d/

chmod +x/etc/rc.d/init.d/keepalived #添加执行权限

Chkconfig keepalived on #设置开机启动

Service keepalived Start #启动

Service keepalived Stop #关闭

Service keepalived Restart #重启

Iv. Configuration of Keepalived

Cp/etc/keepalived/keepalived.conf/etc/keepalived/keepalived.conf-bak

Vi/etc/keepalived/keepalived.conf #编辑, modified to the following code

#########################################################

#以下为192.168.21.129 server:

! Configuration File for Keepalived

Global_defs {

Notification_email {

Acassen@firewall.loc

Failover@firewall.loc

Sysadmin@firewall.loc

}

Notification_email_from Alexandre.Cassen@firewall.loc

Smtp_server 192.168.200.1

Smtp_connect_timeout 30

router_id Lvs_devel

}

Vrrp_script Chk_haproxy {

Script "/etc/keepalived/check_haproxy.sh" #HAproxy服务监控脚本

Interval 2

Weight 2

}

Vrrp_instance Vi_1 {

State MASTER

Interface eth0

VIRTUAL_ROUTER_ID 51

Priority 100

Advert_int 1

Authentication {

Auth_type Pass

Auth_pass 1111

}

Track_script {

Chk_haproxy #监测haproxy进程状态

}

virtual_ipaddress {

192.168.21.253

}

Notify_master the ARP record of the "/etc/keepalived/clean_arp.sh 192.168.21.253" #更新虚拟服务器 (VIP) address to the gateway

}

Vrrp_instance Vi_2 {

State BACKUP

Interface eth0

VIRTUAL_ROUTER_ID 52

Priority 99

Advert_int 1

Authentication {

Auth_type Pass

Auth_pass 1111

}

virtual_ipaddress {

192.168.21.254

}

Notify_master the ARP record of the "/etc/keepalived/clean_arp.sh 192.168.21.254" #更新虚拟服务器 (VIP) address to the gateway

}

#########################################################

: wq! #保存退出

#########################################################

#以下为192.168.21.130 server:

192.168.21.130

! Configuration File for Keepalived

Global_defs {

Notification_email {

Acassen@firewall.loc

Failover@firewall.loc

Sysadmin@firewall.loc

}

Notification_email_from Alexandre.Cassen@firewall.loc

Smtp_server 192.168.200.1

Smtp_connect_timeout 30

router_id Lvs_devel

}

Vrrp_script Chk_haproxy {

Script "/etc/keepalived/check_haproxy.sh" #HAproxy服务监控脚本

Interval 2

Weight 2

}

Vrrp_instance Vi_1 {

State BACKUP

Interface eth0

VIRTUAL_ROUTER_ID 51

Priority 99

Advert_int 1

Authentication {

Auth_type Pass

Auth_pass 1111

}

Track_script {

Chk_haproxy #监测haproxy进程状态

}

virtual_ipaddress {

192.168.21.253

}

Notify_master the ARP record of the "/etc/keepalived/clean_arp.sh 192.168.21.253" #更新虚拟服务器 (VIP) address to the gateway

}

Vrrp_instance Vi_2 {

State MASTER

Interface eth0

VIRTUAL_ROUTER_ID 52

Priority 100

Advert_int 1

Authentication {

Auth_type Pass

Auth_pass 1111

}

virtual_ipaddress {

192.168.21.254

}

Notify_master the ARP record of the "/etc/keepalived/clean_arp.sh 192.168.21.254" #更新虚拟服务器 (VIP) address to the gateway

}

#########################################################

System Yun-wei Www.111cn.net warm reminder: qihang01 original content copyright, reproduced please indicate the source and the original link

: wq! #保存退出

V. Set up Haproxy Service monitoring script

Vi/etc/keepalived/check_haproxy.sh #编辑, add the following code

#########################################################

#!/bin/sh

If [$ (ps-c haproxy--no-header | wc-l)-EQ 0]; Then

/etc/rc.d/init.d/haproxy start

Fi

Sleep 2

If [$ (ps-c haproxy--no-header | wc-l)-EQ 0]; Then

/etc/rc.d/init.d/keepalived stop

Fi

#########################################################

: wq! #保存退出

chmod +x/etc/keepalived/check_haproxy.sh #添加执行权限

Vi. setting ARP records to update virtual server (VIP) addresses to the gateway script

Vi/etc/keepalived/clean_arp.sh #编辑, add the following code

#!/bin/sh

Vip=$1

gateway=192.168.21.2 #网关地址

/sbin/arping-i eth0-c 5-s $VIP $GATEWAY &>/dev/null

: wq! #保存退出

chmod +x/etc/keepalived/clean_arp.sh #添加脚本执行权限

Seven, System kernel optimization

Execute the following command on the two HAPROXY servers separately

echo 1024 60999 >/proc/sys/net/ipv4/ip_local_port_range

echo >/proc/sys/net/ipv4/tcp_fin_timeout

echo 4096 >/proc/sys/net/ipv4/tcp_max_syn_backlog

echo 262144 >/proc/sys/net/ipv4/tcp_max_tw_buckets

echo 262144 >/proc/sys/net/ipv4/tcp_max_orphans

echo >/proc/sys/net/ipv4/tcp_keepalive_time

Echo 1 >/proc/sys/net/ipv4/tcp_tw_recycle

echo 0 >/proc/sys/net/ipv4/tcp_timestamps

echo 0 >/PROC/SYS/NET/IPV4/TCP_ECN

Echo 1 >/proc/sys/net/ipv4/tcp_sack

echo 0 >/proc/sys/net/ipv4/tcp_dsack

Part II: Test haproxy+keepalived whether the normal operation

First, open the Haproxy monitoring page

Http://bbs.111cn.net/haproxy-status

Enter user name, password

Admin

123456

After you sign in, the following figure shows

Second, bbs.111cn.net parsing to 192.168.21.253;sns.111cn.net resolution to 192.168.21.254;

Executing commands on two haproxy servers: 192.168.21.129, 192.168.21.130: IP addr

As shown in the following illustration:

It can be seen now that vip:192.168.21.253 points to 192.168.21.129;vip:192.168.21.254 point 192.168.21.130;

Open in Browser

http://bbs.111cn.net/

http://sns.111cn.net/

As shown in the following illustration:

At this time, BBS and SNS domain name are balanced to 192.168.21.127 above

Third, stop the 192.168.21.127 above Nginx service

Service Nginx Stop

Continue to open the two URLs above, as shown in the following illustration:

At this time, BBS and SNS domain name are balanced to 192.168.21.128 above (because 192.168.21.127 server Nginx service is shut down, realize failover)

Four, close the 192.168.21.129 haproxy Service

Service Haproxy Stop

At this point, execute the command on two Haproxy servers: 192.168.21.129, 192.168.21.130: IP addr

As shown in the following illustration:

System Yun-wei Www.111cn.net warm reminder: qihang01 original content copyright, reproduced please indicate the source and the original link

It can be seen that vip:192.168.21.253 and 192.168.21.254 all point to 192.168.21.130;

At this point, open http://bbs.111cn.net/as shown in the following illustration:

can be accessed normally

V. Restore the Haproxy service above the 192.168.21.129 and restore the Nginx service above 192.168.21.127

Stop the keepalived service 192.168.21.130 above

Service keepalived Stop

Executing commands on two haproxy servers: 192.168.21.129, 192.168.21.130: IP addr

As shown in the following illustration:

It can be seen that vip:192.168.21.253 and 192.168.21.254 all point to 192.168.21.129;

At this point, open http://sns.111cn.net/as shown in the following illustration:

can be accessed normally

Note:

To view the Haproxy log file:

Tail-f/var/log/haproxy.log