Haproxy load Balancing +keepalived high-availability web clusters

First, HAProxy 1.haproxy Introduction

• HAProxy is a free, fast, and reliable solution that provides high availability, load balancing, and proxy software based on TCP (layer fourth) and HTTP (layer seventh) applications that support virtual hosting.

• Haproxy is a free and open source software written in C language

• Haproxy is especially useful for Web sites that are heavily loaded, and often require session-hold or seven-tier processing. The haproxy runs on the current hardware and can support tens of thousands of concurrent connections. and its operating mode makes it easy and safe to integrate into your current architecture, while protecting your Web server from being exposed to the web.

• Haproxy implements an event-driven, single-process model that supports very large number of concurrent connections. A multi-process or multithreaded model is rarely capable of handling thousands of concurrent connections because of memory limitations, System scheduler restrictions, and ubiquitous lock limits. The event-driven model does not have these problems because it implements all of these tasks in a user space (User-space) that has better resource and time management. The disadvantage of this model is that, on multicore systems, these programs often have poor extensibility. That's why they have to be optimized so that each CPU time slice (Cycle) does more work.

• HAProxy support for connection rejection: because the overhead of maintaining a connection is very low, sometimes we need to limit the xxx worm (attack bots), which means restricting their connection opening to limit their harm. This has been developed for a Web site that has been trapped in a small ddosxxx and has been saved

• HAProxy supports fully transparent proxies (typical features of a hardware firewall): You can connect back-end servers with a client IP address or any other address. This feature is only available after the Linux 2.4/2.6 kernel has patched the cttproxy patch. This feature also makes it possible for a particular server to handle part of the traffic without modifying the server's address.

• Well-known websites, including GitHub, BitBucket, Stack Overflow, Reddit, Tumblr, Twitter and Tuenti, and the Amazon Web Services system all use Haproxy.

• Haproxy is a popular cluster scheduling tool at present, similar cluster scheduling tool has many, such as LVS and Nginx, compared to the best performance of LVS, but the construction of relatively complex, Nginx upstream module support cluster function, but the cluster node health Check function is not strong, Performance is not haproxy good.

Cons: Software that only supports TCP and HTTP

Haproxy Official website: http://www.haproxy.org/

2. Load balancing common scheduling algorithm

LVS, Haproxy, nginx the most commonly used scheduling algorithm has three, as described below

• RR (Round Robin). The RR algorithm is the simplest and most commonly used algorithm, that is, polling scheduling. For example, there are three nodes a, B, C, the first user access is assigned to Node A, the second user access is assigned to node B, the third user access is assigned to node C, and the fourth user access continues to be assigned to Node A, polling the allocation access request for load balancing effects. This algorithm also has a weighted poll, that is, the allocation of access requests based on the weight polling of each node

• LC (Least Connections). The LC algorithm, the minimum number of connections, dynamically allocates front-end requests based on the size of the node connections in the backend. For example, there are three nodes a, B, C, the number of connections for each node is a:4, B:5, C:6, at this time if there is a first user connection request, will be assigned to a, the number of connections to A:5, B:5, C:6; the second user request continues to be assigned to a, the number of connections becomes a:6, B:5, C:6 A new request is then configured to B, each time assigning a new request to the node server with the least number of connections. As the actual situation of a, B, C, the number of connections will be released dynamically, it is difficult to show the same number of connections, so this algorithm compared to the RR algorithm has a great improvement, is currently used to more than one algorithm

• SH (Source Hashing). SH is based on the source access scheduling algorithm, this algorithm is used for some session sessions recorded on the server side of the scene, can be based on the source of IP, cookies, etc. to do the cluster scheduling. For example, using the cluster scheduling algorithm based on the source IP, there are three nodes a, B, C, the first user first access is assigned to a, the second user first access is assigned to B, when the first user will be assigned to a second visit, the second user will still be assigned to the second access to B, As long as the load scheduler does not restart, the first user access will be assigned to a, and the second user access will be assigned to B to implement the scheduling of the cluster. The advantage of this scheduling algorithm is to achieve session retention, but some IP traffic is very large when the load is unbalanced, some of the node access is very large, affecting business use

3.Haproxy of logs

• The Haproxy log is output to the syslog of the system by default, it is not very convenient to view, in order to better manage the Haproxy log, we are generally defined separately in the production environment, the method defined is described as follows

• Modify the options for the log configuration in the Haproxy configuration file, and modify the configuration to the following

  log  /dev/log    local0  infolog  /dev/log    local0  notice

  These two lines are configured to modify the original log 127.0.0.1 configuration, mainly to log haproxy info and notice logs to separate log files.
  Then restart Haproxy to complete the Haproxy configuration

• Modifying the Rsyslog configuration
  For ease of administration, the configuration under Haproxy is defined independently to haproxy.conf and placed under/etc/rsyslog.d/, and all configuration files under this directory are automatically loaded at rsyslog startup

  [[email protected] ~]# touch /etc/rsyslog.d/haproxy.conf[[email protected] ~]# vim /etc/rsyslog.d/haproxy.conf

Add the following content:

if ($programname == ‘haproxy‘ and $syslogseverity-text == ‘info‘) then -/var/log/haproxy/haproxy-info.log& ~if ($programname == ‘haproxy‘ and $syslogseverity-text == ‘notice‘) then -/var/log/haproxy/haproxy-notice.log& ~

This part of the configuration is to record the Haproxy info log to/var/log/haproxy/haproxy-info.log, notice log to/var/log/haproxy/haproxy-notice.log, where "& ~" means that Rsyslog stops processing this information after the log file is written to it. The syntax configured here is written using the Rainerscript scripting language

• Of course, you can also use another workaround, modify the/etc/rsyslog.conf configuration file, add the following two lines to the configuration file:

  local0.info                                             /var/log/haproxy-info.loglocal0.notice                                           /var/log/haproxy-notice.log

  This can also achieve the same effect above

• Test log information
  After the client accesses http://192.168.1.10/test.html, you can use tail-f/var/log/haproxy/haproxy-info.log to instantly view Haproxy's access request log information

  [[email protected] ~]# tail -f /var/log/haproxy/haproxy-info.log Sep  9 22:30:05 localhost haproxy[2634]: 192.168.1.30:1165 [09/Sep/2015:22:30:02.202] webcluster webcluster/inst2 0/0/0/1/3255 200 256 - - CD-- 0/0/0/0/0 0/0 "GET /test.html HTTP/1.1"

Parameter optimization for 4.Haproxy

For the parameter optimization of Haproxy, several key parameters are listed below, and the optimization suggestions for the production environment of each parameter are described.

Parameters	parameter Description	Optimization Recommendations
Maxconn	Maximum number of connections	This parameter is adjusted according to the actual application, it is recommended to use 10 240
Daemon	Daemon mode	Haproxy can be started using non-daemon mode, the production environment is recommended to start with the daemon mode
Nbproc	Number of concurrent processes with load balancing	Recommended equal or twice times the number of CPU cores for the current server
Retries	Retry Count	This parameter is mainly used to check the cluster node, if the node is many, and the concurrency is large, set to 2 or 3 times, in the case of a few server nodes, can be set to 5 or 6 times
Option Http-server-close	Proactively turn off HTTP request options	It is recommended to use this option in a production environment to avoid the accumulation of HTTP connections due to timeout time setting too long
Timeout http-keep-alive	Long Connection Timeout time	This option sets the long connection time-out period, depending on the application's own feature settings, can be set to 10s
Timeout http-request	HTTP request time-out	It is recommended that this time be set to 5~10s to increase the speed of HTTP connection release
Timeout client	Client timeout time	If the traffic is too large, the node response is slow, you can set this time shorter, it is recommended to set to about 1min

Ii. Deployment of Haproxy+keepalived Environment

Five servers, two web, two haproxy, the network card is set to Vmnet1, the remaining one for the gateway, need two network card, the first block Vmnnet1, the second block bridge

Package

HOST	OS	IP
Web01	Centos6.7_64	192.168.1.30	httpd
Web02	Centos6.7_64	192.168.1.40	httpd
Haproxy+keepalived-1	Centos6.7_64	192.168.1.10	Haproxy, keepalived
Haproxy+keepalived-2	Centos6.7_64	192.168.1.20	Haproxy, keepalived
Gateway	Centos6.7_64	192.168.1.1	Iptables

Deploying Web-1

1. Configure the IP address

vim /etc/sysconfig/network-scrips/ifcfg-eth0DEVICE=eth0TYPE=EthernetONBOOT=yesNM_CONTROLLED=noBOOTPROTO=staticIPADDR=192.168.1.30NETMASK=255.255.255.0GATEWAY=192.168.1.1

/etc/init.d/network restart

2. Configuring the Install Yum

rm -rf /etc/yum.repos.d/*

vim /etc/yum.repos.d/local.repo[local]name=localbaseurl=file:///mntgpgcheck=0

3. Deploying httpd

yum -y install httpd && echo "This is Web1" >/var/www/html/index.html

/etc/init.d/httpd start && chkconfig --level 35 httpd on

Deploying Web-2

1. Configure the IP address

vim /etc/sysconfig/network-scrips/ifcfg-eth0DEVICE=eth0TYPE=EthernetONBOOT=yesNM_CONTROLLED=noBOOTPROTO=staticIPADDR=192.168.1.40NETMASK=255.255.255.0GATEWAY=192.168.1.1

/etc/init.d/network restart

2. Configuring the Install Yum

rm -rf /etc/yum.repos.d/*

vim /etc/yum.repos.d/local.repo[local]name=localbaseurl=file:///mntgpgcheck=0

mount /dev/cdrom /mnt

3. Deploying httpd

yum -y install httpd && echo "This is Web2" >/var/www/html/index.html

/etc/init.d/httpd start && chkconfig --level 35 httpd on

Deploying Haproxy+keepalived-1

1. Configure IP

vim /etc/sysconfig/network-scrips/ifcfg-eth0DEVICE=eth0TYPE=EthernetONBOOT=yesNM_CONTROLLED=noBOOTPROTO=staticIPADDR=192.168.1.10NETMASK=255.255.255.0GATEWAY=192.168.1.1

/etc/init.d/network restart

2. Configure Yum and install dependent packages

rm -rf /etc/yum.repos.d/*

vim /etc/yum.repos.d/local.repo[local]name=localbaseurl=file:///mntgpgcheck=0

mount /dev/cdrom /mnt

yum -y install kernel-devel openssl-devel popt-devel  pcre-devel bzip2-devel

3. Installing keepalived

tar zxvf keepalived-1.2.2.tar.gz -C /usr/src/

cd /usr/src/keepalived-1.2.2/

./configure --prefix=/ --with-kernel-dir=/usr/src/kernels/2.6.32-431.el6.x86_64/

make && make install

chkconfig --add keepalived && chkconfig --level 35 keepalived on

4. Edit the HA configuration file

vim /etc/keepalived/keepalived.confglobal_defs {    router_id HA_TEST_R1    ##本服务器的名称}vrrp_instance VI_1 {        ##定义VRRP热备实例    state MASTER        ##MASTER表示主服务器    interface eth0      ##承载VIP地址的物理接口    virtual_router_id 1     ##虚拟路由器的ID号    priority 100        ##优先级，数值越大优先级越高    advert_int 1        ##通告间隔秒数（心跳频率）    authentication {        ##认证信息        auth_type PASS      ##认证类型        auth_pass 123456    ##密码字串    }    virtual_ipaddress {  192.168.1.254     ##指定漂移地址（VIP）    }}

5. Compile and install Haproxy

tar zxvf haproxy-1.4.24.tar.gz -C /usr/src/

cd /usr/src/haproxy-1.4.24/

make TARGET=linux26                         //linux26代表为kernel2.6的版本(uname -r)

make install

6. Edit the Haproxy configuration file

mkdir /etc/haproxy    //新建软件配置文件目录

cp /usr/src/haproxy-1.4.24/examples/haproxy.cfg /etc/haproxy    //将haproxy.cfg文件复制到配置文件目录

The Haproxy configuration file is typically divided into three parts, namely, Global, defaults, and listen. Global configuration, defaults default configuration, listen configuration for application components

vim/etc/haproxy/haproxy.cfg#\ this config needs haproxy-1.1.28 or Haproxy-1.2.1global log 127.0.0.1 local0 Global log configuration and specify logging level log 127.0.0.1 local1 notice//notice to log level, usually with 24 levels #log Loghost local0 Info maxconn 40                  96//Maximum number of connections UID 99//Run user, default nobody GID 99//Run group, default nobody daemon                Run Haproxy #debug as daemon #quietdefaults log Global//apply Global Log configuration mode HTTP Default mode {tcp|http|health} option Httplog//Enable logging HTTP request option Dontlognull//Start the entry, the log will            does not log a null connection retries 3//Connection back-end server failed reconnection Maxconn 2000//per process maximum number of connections available Contimeout 5000 Connection Timeout clitimeout 50000//Client Timeout srvtimeout 50000//server Time Out listen Webcluster 19 2.168.1.254:80 option Httpchk get/index.html//Check server index.html file balance roundrobin//Develop load balancing algorithm serv ER inst1 192.168.1.30:80 Check Inter Fall 3//define the IP server inst2 192.168.1.40:80 check Inter for the back end server Fall 3 

7. Start the Haproxy service

cp /usr/src/haproxy-1.4.24/examples/haproxy.init /etc/init.d/haproxy

chmod +x /etc/init.d/haproxy

ln -s /usr/local/sbin/haproxy /usr/sbin

mkdir /usr/share/haproxy

/etc/init.d/keepalived start

/etc/init.d/haproxy start

chkconfig --level 35 haproxy on

Deploying Haproxy+keepalived-2

1. Configure IP

vim /etc/sysconfig/network-scrips/ifcfg-eth0DEVICE=eth0TYPE=EthernetONBOOT=yesNM_CONTROLLED=noBOOTPROTO=staticIPADDR=192.168.1.20NETMASK=255.255.255.0GATEWAY=192.168.1.1

/etc/init.d/network restart

2. Configure Yum and install dependent packages

rm -rf /etc/yum.repos.d/*

vim /etc/yum.repos.d/local.repo[local]name=localbaseurl=file:///mntgpgcheck=0

mount /dev/cdrom /mnt

yum -y install kernel-devel openssl-devel popt-devel  pcre-devel bzip2-devel

3. Installing keepalived

tar zxvf keepalived-1.2.2.tar.gz -C /usr/src/

cd /usr/src/keepalived-1.2.2/

./configure --prefix=/ --with-kernel-dir=/usr/src/kernels/2.6.32-431.el6.x86_64/

make &&make install

chkconfig --add keepalived && chkconfig --level 35 keepalived on

4. Edit the HA configuration file

scp [email protected]:/etc/keepalived/keepalived.cfg /etc/keepalived/

vim /etc/keepalived/keepalived.confglobal_defs {    router_id HA_TEST_R2    ##本服务器的名称}vrrp_instance VI_1 {        ##定义VRRP热备实例    state BACKUP        ##MASTER表示主服务器    priority 10         ##优先级，数值越大优先级越高

5. Compile and install Haproxy

tar zxvf haproxy-1.4.24.tar.gz -C /usr/src/

cd /usr/src/haproxy-1.4.24/

make TARGET=linux26

make install

6. Edit the Haproxy configuration file

mkdir /etc/haproxy/

scp [email protected]:/etc/haproxy/haproxy.conf /etc/haproxy/

7. Writing shell scripts

cp /usr/src/haproxy-1.4.24/examples/haproxy.init /etc/init.d/haproxy

chmod +x /etc/init.d/haproxy

ln -s /usr/local/sbin/haproxy /usr/sbin

mkdir /usr/share/haproxy && chkconfig --level 35 haproxy on

vim /root/haproxy.sh#!/bin/bashA=$(netstat -utpln | grep 80 | wc -l)/sbin/ip a | /bin/grep 192.168.1.254 &>/dev/nullif [ $? -eq 0 ] && [ $A -eq 0 ]then  /etc/init.d/haproxy restart  echo "Haproxy启动中"else  /etc/init.d/haproxy stopfi

chmod +x /root/haproxy.sh

crontab -e  */1 * * * *  /root/haproxy.sh

scp /root/haproxy.sh [email protected]:/root

chmod +x /root/haproxy.sh (第一个调度器，192.168.1.10)

crontab -e  */1 * * * *  /root/haproxy.sh (第一个调度器，192.168.1.10)

Gateway

Acting as a router role

1. Configure IP

vim /etc/sysconfig/network-scrips/ifcfg-eth0DEVICE=eth0TYPE=EthernetONBOOT=yesNM_CONTROLLED=noBOOTPROTO=staticIPADDR=192.168.1.1NETMASK=255.255.255.0

cp /etc/sysconfig/network-scrips/ifcfg-eth0 /etc/sysconfig/network-scrips/ifcfg-eth1

vim /etc/sysconfig/network-scrips/ifcfg-eth1DEVICE=eth0TYPE=EthernetONBOOT=yesNM_CONTROLLED=noBOOTPROTO=dhcp

/etc/init.d/network restart

2. Turn on route forwarding

vim /etc/sysctl.conf 7 net.ipv4.ip_forward = 1

sysctl -p

3. Writing firewall rules

/etc/init.d/iptables stop

iptables -t nat -I PREROUTING -d 192.168.10.139 -p tcp --dport 80 -j DNAT --to-destination 192.168.1.254:80

/etc/init.d/iptables save && chkconfig --level 35 iptables on

4. Access Testing

ie→192.168.10.139

Can test the scheduler for high availability and load balancing of web clusters

Reference URL:
http://blog.51cto.com/painting/1693259
http://www.ttlsa.com/linux/haproxy-study-tutorial/
https://baike.baidu.com/item/haproxy/5825820

Haproxy load Balancing +keepalived high-availability web clusters