LVS + heartbeat implementation case and experience sharing

It O & M Expert Network
Http://www.linuxtone.org
LVS + heartbeat implementation case and experience sharing
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning)

You are welcome to reprint the document. During reprinting, please be sure to mark the original source and author information of the article as hyperlinks and this statement.
1. Basic Introduction
Background description
Because the business volume of a large commercial website is growing rapidly, the current website architecture cannot alleviate users' access.
Stress and spof. A high-availability server Load balancer cluster solution is urgently needed
Case, http://www.linuxTone.org (it O & M Expert Network) according to the situation of the Business Network, the deployment of a set of negative
Load Balancing and highly available cluster solutions and Related Optimization. After the implementation of the project, the commercial website receives the Web pressure
And the access speed is greatly improved. The following is the relevant configuration documents to share the hope with friends.
It is helpful for friends who want to learn more about LVS/ha.
Features
1. Introduce the two installation methods of LVS + heartbeat
2. Use heartbeat 2.x style configuration (using CIB. XML)
3. Test log analysis and related skills
4. share two experiences in real cases
 
2. network topology and related descriptions
 
1. Host Name: (/etc/hosts)
192.168.1.102 primarylb.linuxtone.org
192.168.1.103 backuplb.linuxtone.org
2. VIP [eth0: 0] 192.168.1.106
Realserver1 192.168.1.104
Realserver2 192.168.1.105
3. heartbeat
A) Use a serial port direct connection cable (Serial null modem cable) heartbeat
B) eth1: 10.0.0.1 10.0.0.2 (cross-line direct connection to heartbeat)
 
Iii. Install heartbeat [Yum installation, compilation and installation]
1. Install heartbeat through yum
# Yum install heartbeat-* // install ha
# Yum install heartbeat-ldirectord // install ldirectord to achieve high availability of LVS
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com it O & M Expert Network
Http://www.linuxtone.org
# Rpm-Qa | grep heartbeat // view the installation package
Heartbeat-stonith-2.1.3-3.el5.centos
Heartbeat-devel-2.1.3-3.el5.centos
Heartbeat-pils-2.1.3-3.el5.centos
Heartbeat-2.1.3-3.el5.centos
Heartbeat-ldirectord-2.1.3-3.el5.centos
Heartbeat-gui-2.1.3-3.el5.centos
 
# Rpm-Q heartbeat-D // view the relevant doc directory of heartbeat
Copy the configuration file to the/etc/ha. d directory.
# Cp/usr/share/doc/heartbeat-2.1.3/ha. Cf/etc/ha. d/
# Cp/usr/share/doc/heartbeat-2.1.3/authkeys/etc/ha. d/
# Cp/usr/share/doc/heartbeat-2.1.3/haresources/etc/ha. d
 
 
2. Compile and install heartbeat
Download related software packages:
Http://linux-ha.org/DownloadSoftware

Http://www.packetfactory.net/libnet/dist/libnet.tar.gz

Compile and install
# Tar-xvzf libnet.tar.gz
# Cd Libnet
#./Configure
# Make; make install
 
# Groupadd-G 694 haclient
# Useradd-u 694-G haclient hacluster
 
# Tar-xvzf heartbeat-2.1.3.tar.gz
# Cdheartbeat-2.1.3
#./Configureme configure
# Make; make install
 
# Cp DOC/ha. Cf DOC/haresources DOC/authkeys/etc/ha. d/
# Cp ldirectord/ldirectord. Cf/etc/ha. d/
 
Note]
The heartbeat package can be found in the RHEL or centos installation disk. Required software packages: heartbeat, Libnet,
Net-snmp, net-snmp-libs, Perl-compress-zlib, Perl-HTML-parser, Perl-HTML-tagset,
Perl-libwww-perl, Perl-mailinuxtoneools, Perl-TimeDate, Perl-Uri, and ipvsadm,
Some are on the rhel5 disc, and some are on the Internet.
 
 
4. Configure heartbeat to achieve high service availability.
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com it O & M Expert Network
Http://www.linuxtone.org
1. Configure two ha host names respectively.
# Vi/etc/hosts
192.168.1.102 primarylb.linuxtone.org
192.168.1.103 backuplb.linuxtone.org
 
2./etc/ha. d/ha. cf configuration
# Cd/etc/ha. d
# Cat ha. Cf | grep-V ^ #
Logfile/var/log/ha-Log
Logfacility local0
Keepalive 2
Deadtime 30
Warntime 10
Initdead 120
Udpport 694
Baud 19200
Serial/dev/ttys0 # Linux
# Ucast eth2 10.0.0.3
Ucast eth0 192.168.1.103
Auto_failback on
Watchdog/dev/Watchdog
CRM On
Node primarylb.linuxtone.org
Node backuplb.linuxtone.org
Ping 192.168.1.103
Ping_group lbgroup 10.0.0.1 10.0.0.2
Ping_group lbgroup 192.168.1.102 192.168.1.103
Respawn hacluster/usr/lib/heartbeat/ipfail
Apiauth ipfail gid = haclient uid = hacluster
 
3./etc/ha. d/authkeys
# Vi authkeys
Auth 1
1 sha1 hey!
# Chmod 600 authkeys
 
4./etc/ha. d/haresources
# Cat haresources | grep-V "#"
Primarylb.linuxtone.org LVS ldirectord
 
5. Establish a soft connection to allow ha to automatically take over resources such as lvs-dr.sh and ldirectord.
# Ln-S/etc/init. d/LVS/etc/ha. d/resource. d // etc/init. d/LVS store LVS start related script Century

# Ln-S/usr/sbin/ldirectord/etc/ha. d/resource. d
 
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com it O & M Expert Network
Http://www.linuxtone.org
6. Use heartbeat 2.x style and CRM to manage resources.
#/Usr/lib/heartbeat/haresources2cib. py-stout-C/etc/ha. d/ha. Cf/etc/ha. d/haresources

Run the generated CIB. xml file in the/var/lib/heartbeat/CRM directory.
Note]
About CRM (cluster resource manager): the CRM resource configuration method is introduced in the new version. If
To use this format to configure resources (IP address, service, shared disk, etc.), refer to the configuration example:
Http://www.linux-ha.org/v2/Examples/Simple. Functions of the/var/lib/heartbeat/CRM/CIb. xml file and
/Etc/ha. d/haresources is the same, which is the relationship between the two.
# Cat/dev/null> haresources // clear haresources and use CRM to manage resources
 
7./etc/ha. d/ldirectord. cf configuration
# Global ctictives
Checktimeout = 3
Checkinterval = 1
Fallback = 127.0.0.1: 80
Autoreload = Yes
Logfile = "/var/log/ldirectord. log"
# Logfile = "local0"
Emailalert = "cnseek@gmail.com"
Emailalertfreq= 3600
Emailalertstatus = all
Quiescent = No
 
# Sample for an HTTP virtual service
Virtual = 192.168.1.106: 80
Real = 192.168.1.104: 80 Gate
Real = 192.168.1.105: 80 Gate
# Fallback = 127.0.0.1: 80 Gate
# Service = http
# Request = "index.html"
# Receive = "test page"
# Virtualhost = some.domain.com. au
Scheduler = WRR
Persistent = 5
# Netmask = 255.255.255.255.255
Protocol = TCP
Checktype = connect
Checkport = 80
# Request = "index.html"
# Receive = "test page"
# Virtualhost = www. x. y. Z
Note]
Ldirectord is used to monitor the real server. When the Real Server fails, it is removed from the Load balancer list.
Delete, restore, and re-Add. When heartbeat is installed, ldirectord has been installed. quiescent = no in the configuration.
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com it O & M Expert Network
Http://www.linuxtone.org
Set
If the RealServer machine is down, it will be immediately deleted from the ipvs list and re-added Upon recovery.
 
8. Start the heartbeat Service
#/Etc/init. d/heartbeat start
 
5. Test heartbeat and Log Analysis
1. test whether the serial line configuration is normal
A serial connection connects two nodes through a serial port using a direct connection cable. Now test the serial connection,
Enter the following content on primarylb (receiver:
Cat </dev/ttys0
Enter the following content on backuplb (sender:
Echo "serial connection test! Welcome http://www.linuxtone.org! ">/Dev/ttys0

 
2. If the master node needs to be maintained, You can first transfer it to the backup server.
/Usr/lib/heartbeat/hb_standby (hb_standby Command forces the resource to be migrated to the backup server)
If the master node is to be switched back to take charge of the Service:
/Usr/lib/heartbeat/hb_takeover
 
3. Unplug the network cable and test the switchover after the network of the primary node is disconnected.
4. Stop HEARTBEAT (/etc/init. d/heartbeat stop) of primary to check whether backup can take over resources normally.
5. shutdown the primary host and test whether the switchover is normal. Basically, it is similar to test 2 above.
Note]
# Tail-F/var/log/messages // check the log changes on the two Server Load balancer servers to troubleshoot the error.
 
Vi. Precautions and references

1. Problems Encountered during project implementation and summary. [experience sharing]
[The following are two problems encountered in the project. It is helpful to ask for help]
A) if the standby relationship is used between two vs, when one vs takes over the LVS service, the Network may
The Mac cache table of the vro cannot refresh the MAC address in time.
The address is the MAC address of the replaced vs. There are two solutions: one is to modify the MAC address of the new Vs and the other
The send_arp/arpiing command is used. The arping command is used as an example.
/Sbin/arping-I eth0-C 3-S $ {VIP }$ {gateway_ip}>/dev/null 2> & 1
For example:
/Sbin/arping-I eth0-C 3-s 192.168.1.6 192.168.1.1

B) after a RealServer service is down, how can I automatically delete and restore from the ipvs list?
Configure ldirectord. conf
Quiescent = No or Echo 1>/proc/sys/NET/IPv4/VS/expire_nodest_conn
 
2. Ha. Cf comments
When auto_failback is set to on, all resources will be retrieved from the slave node once the master node recovers online again.
The stonith device is a device that can automatically turn off the power to respond to software commands.
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com it O & M Expert Network
Http://www.linuxtone.org
Watchdog can be either a hardware circuit or a software timer in implementation, and can be automatically duplicated when the system fails.
Start the system.
 
3. Check the LVS status
The operation steps are detailed at the command line level.
View LVS connection information: ipvsadm-l-N
Check the LVS throughput: ipvsadm-l-N-Rate
View LVS statistics: ipvsadm-l-N-stat
View LVS connection status changes in real time: Watch ipvsadm-ln
 
4. Attach the LVS (DR) script
#! /Bin/bash
# Description: Start LVS of directorserver
# Written by: netseek http://www.linuxtone.org

GW = 192.168.1.1

# Website Director VIP.
Web_vip = 192.168.1.106
Web_rip1 = 192.168.1.104
Web_rip2 = 192.168.1.105
 

./Etc/rc. d/init. d/functions

Logger $0 called with $1

Case "$1" in

Start)
# Clear all iptables rules.
/Sbin/iptables-F
# Reset iptables counters.
/Sbin/iptables-z
# Clear all rules SADM rules/services.
/Sbin/ipvsadm-C

# Set lvs vip for Dr
/Sbin/ipvsadm -- set 30 5 60
/Sbin/ifconfig eth0: 0 $ web_vip broadcast $ web_vip netmask 255.255.255.255

Up
/Sbin/route add-host $ web_vip Dev eth0: 0
/Sbin/ipvsadm-a-t $ web_vip: 80-s WRR-P 3
/Sbin/ipvsadm-a-t $ web_vip: 80-r $ web_rip1: 80-g-W 1
/Sbin/ipvsadm-a-t $ web_vip: 80-r $ web_rip2: 80-g-W 1
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com it O & M Expert Network
Http://www.linuxtone.org
Touch/var/lock/subsys/ipvsadm>/dev/null 2> & 1

# Set ARP
/Sbin/arping-I eth0-C 5-S $ web_vip $ Gw>/dev/null 2> & 1
;;
Stop)
/Sbin/ipvsadm-C
/Sbin/ipvsadm-z
Ifconfig eth0: 0 down
Route del $ web_vip>/dev/null 2> & 1
Rm-RF/var/lock/subsys/ipvsadm>/dev/null 2> & 1
/Sbin/arping-I eth0-C 5-S $ web_vip $ GW
Echo "ipvsadm stoped"
;;

Status)

If [! -E/var/lock/subsys/ipvsadm]; then
Echo "ipvsadm is stoped"
Exit 1
Else
Ipvsadm-ln
Echo "...... wait SADM is OK ."
Fi
;;

*)
Echo "Usage: $0 {START | stop | status }"
Exit 1
Esac

Exit 0
 
5. Detailed description of the ldirectord. cf file:
Http://www.linuxtone.org/thread-1388-1-1.html
LVS high availability solution Summary: http://www.linuxtone.org/thread-1402-1-1.html [recommended]

LVS Load Balancing Data Summary: http://www.linuxtone.org/thread-1191-1-1.html [recommended]

For more information, see: http://www.linuxtone.org/forum-26-1.html

 
6. Contact Us [if you have any questions, please contact us]:
It O & M expert network http://www.linuxtone.org mail:
Cnseek@gmail.com
 
Author: netseek http://www.linuxtone.org (it O & M Expert Network | cluster architecture | Performance Tuning) MSN: cnseek@msn.com