Configure httpd High Availability Service Based on Heartbeat in Linux

Configure httpd High Availability Service Based on Heartbeat in Linux

Heartbeat is an open-source Linux-based, widely used high-availability cluster system. We can build a web high-availability service environment based on Heartbeat. This article provides a simple example in CentOS 6.5 and analyzes its logs for your reference.

For more information about Heartbeat, see:

Heartbeat cluster component Overview

Heartbeat installation and configuration

1. Configure host resolution and network
### Host name configuration, consistent with the resolution configuration in/etc/hosts
[Root @ orasrv1 ~] # More/etc/sysconfig/network
NETWORKING = yes
HOSTNAME = orasrv1.xlk.com

[Root @ orasrv1 ~] # More/etc/hosts

127.0.0.1 localhost. localdomain localhost

# Public ip
192.168.21.10 orasrv1.xlk.com orasrv1 ### node 1
192.168.21.13 orasrv2.xlk.com orasrv2 ### node 2

# Private ip
192.168.144.128 orasrv1-priv.xlk.com orasrv1-priv
192.168.144.129 orasrv2-priv.xlk.com orasrv2-priv

[Root @ orasrv1 ~] # Scp/etc/hosts orasrv2:/etc/hosts

[Root @ orasrv1 ~] # Ifconfig | grep eth-A1
Eth0 Link encap: Ethernet HWaddr 00: 0C: 29: 35: 31: E5 # eth0 provides external services
Inet addr: 192.168.21.10 Bcast: 192.168.21.255 Mask: 255.255.255.0
--
Eth1 Link encap: Ethernet HWaddr 00: 0C: 29: 35: 31: EF # eth1 is used as the heartbeat Network
Inet addr: 192.168.144.128 Bcast: 192.168.144.255 Mask: 255.255.255.0

Ii. Configuration equivalence
### Node 1
# Ssh-keygen-t rsa-f ~ /. Ssh/id_rsa-p''
# Ssh-copy-id-I./. ssh/id_rsa.pub root@192.168.21.13
# Ssh orasrv2 date; ssh 192.168.21.13 date;

### Node 2
# Ssh-keygen-t rsa-f ~ /. Ssh/id_rsa-p''
# Ssh-copy-id-I./. ssh/id_rsa.pub root@192.168.21.10
# Ssh orasrv1 date; ssh 192.168.21.10 date;

3. Configure Time Synchronization
Here. Refer to: configure the Linux 11G R2 rac ntp service

4. Install httpd
Node 1: orasrv1
# Yum install httpd
# Service httpd start ### start a service
# Echo "orasrv1.xlk.com">/var/www/html/index.html
### Open a browser and check whether access is normal. http: // 192.168.21.10/index.html
### If the service can be accessed normally, close the service and disable auto-start
# Service httpd stop
# Chkconfig httpd off

Node 2: orasrv2
# Yum install httpd
# Service httpd start ### start a service
# Echo "orasrv2.xlk.com">/var/www/html/index.html
### Open a browser and check whether access is normal. http: // 192.168.21.13/index.html
### If the service can be accessed normally, close the service and disable auto-start
# Service httpd stop
# Chkconfig httpd off

5. Configure heartbeat
1. Configure the authentication method
[Root @ orasrv1 ~] # Vi/etc/ha. d/authkeys | grep-v ^ #
Auth 1
1 md5 74a6a503b5bec9373b22bb630608df41 ### use md5 for verification

[Root @ orasrv1 ~] # Chmod 600/etc/ha. d/authkeys

2. Configure ha. cf
[Root @ orasrv1 ~] # Vi/etc/ha. d/ha. cf
Logfile/var/log/ha-log
Keepalive 2
Deadtime 15
Warntime 10
Initdead 60
Udpport 694
Bcast eth1
Auto_failback off
Node orasrv1.xlk.com
Node orasrv2.xlk.com
Ping 192.168.21.254
# Respawn hacluster/usr/lib64/heartbeat/ipfail

3. Configure haresources
[Root @ orasrv1 ~] # Vi/etc/ha. d/haresources
Orasrv1.xlk.com IPaddr: 192.168.21.1/24/eth0 httpd

4. Synchronize to the second node
# Scp-p/etc/ha. d/{ha. cf, authkeys, haresources} 192.168.21.13:/etc/ha. d/

Vi. Verification Test
1. Start heartbeat in node 1
### Note: This example does not use a shared file system, such as configuring drbd or nfs.
[Root @ orasrv1 ~] # Service heartbeat start
Starting High-Availability services:
11:46:51 INFO: Resource is stopped ### the system prompts that the Resource is in the stopped status.
Done.

### Log Analysis on node 1
[Root @ orasrv1 ~] # Tail-fn 50/var/log/ha-log
Heartbeat [4373]: 2015/11/06 _ 11:46:51 info: Version 2 support: false
Heartbeat [4373]: 2015/11/06 _ 11:46:51 WARN: Logging daemon is disabled -- enabling logging daemon is recommended
Heartbeat [4373]: _ 11:46:51 info :**************************
Heartbeat [4373]: 2015/11/06 _ 11:46:51 info: Configuration validated. Starting heartbeat 2.1.4
Heartbeat [4374]: 2015/11/06 _ 11:46:51 info: heartbeat: version 2.1.4
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: Heartbeat generation: 1446714158
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0-Status: 1
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: glib: ping heartbeat started.
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: G_main_add_TriggerHandler: Added signal manual handler
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: G_main_add_TriggerHandler: Added signal manual handler
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: G_main_add_SignalHandler: Added signal handler for signal 17
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: Local status now set to: 'up'
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: Link 192.168.21.254: 192.168.21.254 up.
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: Status update for node 192.168.21.254: status ping
Heartbeat [4374]: 2015/11/06 _ 11:46:52 info: Link orasrv1.xlk.com: eth0 up.
Heartbeat [4374]: 2015/11/06 _ 11:47:52 WARN: node orasrv2.xlk.com: is dead
### The preceding log shows that the heartbeat on node 1 has been successfully started and resources have not been allocated yet.
### The following prompt shows that node 2 is in the dead state. The interval is 60 s (46: 52s ~ 47: 52 s), that is, initdead in our configuration file.
### That is to say, wait for 1 minute until the heartbeat of the other side is detected. In this case, node 2 is considered dead. Note that the death here is not the same as deadtime.

### Start resource allocation below
Heartbeat [4374]: 2015/11/06 _ 11:47:52 info: Comm_now_up (): updating status to active
Heartbeat [4374]: 2015/11/06 _ 11:47:52 info: Local status now set to: 'active'
Heartbeat [4374]: 2015/11/06 _ 11:47:52 WARN: No STONITH device configured.
Heartbeat [4374]: 2015/11/06 _ 11:47:52 WARN: Shared disks are not protected.
Heartbeat [4374]: 2015/11/06 _ 11:47:52 info: Resources being acquired from orasrv2.xlk.com.
Harc [4385]: _ 11:47:52 info: Running/etc/ha. d/rc. d/status
Mach_down [4419]: 2015/11/06 _ 11:47:52 info:/usr/share/heartbeat/mach_down: nice_failback: foreign resources acquired
Mach_down [4419]: 2015/11/06 _ 11:47:52 info: mach_down takeover complete for node orasrv2.xlk.com.
Heartbeat [4374]: 2015/11/06 _ 11:47:52 info: Initial resource acquisition complete (T_RESOURCES (us ))
Heartbeat [4374]: _ 11:47:52 info: mach_down takeover complete.
IPaddr [4457]: 2015/11/06 _ 11:47:52 INFO: Resource is stopped
Heartbeat [4386]: 2015/11/06 _ 11:47:52 info: Local Resource acquisition completed.
Harc [4518]: 2015/11/06 _ 11:47:52 info: Running/etc/ha. d/rc. d/ip-request-resp
Ip-request-resp [4518]: 2015/11/06 _ 11:47:52 received ed ip-request-resp IPaddr: 192.168.21.1/24/eth0 OK yes
ResourceManager [4537]: 2015/11/06 _ 11:47:52 info: Acquiring resource group: orasrv1.xlk.com IPaddr: 192.168.21.1/24/eth0 httpd
IPaddr [4563]: 2015/11/06 _ 11:47:52 INFO: Resource is stopped ### the previous log obtains the Resource group, and the next log starts the Resource.
ResourceManager [4537]: 2015/11/06 _ 11:47:52 info: Running/etc/ha. d/resource. d/IPaddr 192.168.21.1/24/eth0 start
IPaddr [4658]: 2015/11/06 _ 11:47:53 INFO: Using calculated netmask for 192.168.21.1: 255.255.255.0
IPaddr [4658]: 2015/11/06 _ 11:47:53 INFO: eval ifconfig eth0: 0 192.168.21.1 netmask 255.255.255.0 broadcast 192.168.21.255
IPaddr [4629]: 2015/11/06 _ 11:47:53 INFO: Success ### successful vip allocation
ResourceManager [4537]: 2015/11/06 _ 11:47:53 info: Running/etc/init. d/httpd start ### enable the httpd service
Heartbeat [4374]: 2015/11/06 _ 11:48:03 info: Local Resource acquisition completed. (none)
Heartbeat [4374]: 2015/11/06 _ 11:48:03 info: local resource transition completed.

### Verify the VIP allocation, as shown in the following figure. The VIP has been obtained on eth0: 0.
[Root @ orasrv1 ~] # Ifconfig | grep eth-A1
Eth0 Link encap: Ethernet HWaddr 00: 0C: 29: 35: 31: E5
Inet addr: 192.168.21.10 Bcast: 192.168.21.255 Mask: 255.255.255.0
--
Eth0: 0 Link encap: Ethernet HWaddr 00: 0C: 29: 35: 31: E5
Inet addr: 192.168.21.1 Bcast: 192.168.21.255 Mask: 255.255.255.0
--
Eth1 Link encap: Ethernet HWaddr 00: 0C: 29: 35: 31: EF
Inet addr: 192.168.144.128 Bcast: 192.168.144.255 Mask: 255.255.255.0

### Verify the httpd service. In this case, access the home page through VIP and return data from orasrv1, that is, node 1.
Http: // 192.168.21.1/index.html
Orasrv1.xlk.com

2. Start heartbeat on node 2
[Root @ orasrv2 ~] # Service heartbeat start
Starting High-Availability services:
11:50:22 INFO: Resource is stopped
Done.

### The following logs are Node 1, and the logs of Node 1 are tracked.
### From the following logs, we can know that orasrv2.xlk.com: status up and remote resource transition completed
Heartbeat [4374]: 2015/11/06 _ 11:50:23 info: Link orasrv2.xlk.com: eth0 up.
Heartbeat [4374]: 2015/11/06 _ 11:50:23 info: Status update for node orasrv2.xlk.com: status init
Heartbeat [4374]: 2015/11/06 _ 11:50:23 info: Status update for node orasrv2.xlk.com: status up
Harc [4809]: _ 11:50:23 info: Running/etc/ha. d/rc. d/status
Harc [4825]: _ 11:50:23 info: Running/etc/ha. d/rc. d/status
Heartbeat [4374]: 2015/11/06 _ 11:50:25 info: Status update for node orasrv2.xlk.com: status active
Harc [4840]: _ 11:50:25 info: Running/etc/ha. d/rc. d/status
Heartbeat [4374]: 2015/11/06 _ 11:50:25 info: remote resource transition completed.

3. Node 1 stops heartbeat
### Use the following command to stop heartbeat on node 1
[Root @ orasrv1 ~] # Service heartbeat stop
Stopping High-Availability services:
Done.

### Observe the Log Changes of Node 1 at this time
Heartbeat [4374]: 2015/11/06 _ 11:53:13 info: Heartbeat shutdown in progress. (4374)
Heartbeat [4873]: 2015/11/06 _ 11:53:13 info: Giving up all HA resources. # Node 1 begins to discard resources
ResourceManager [4886]: 2015/11/06 _ 11:53:14 info: Releasing resource group: orasrv1.xlk.com IPaddr: 192.168.21.1/24/eth0 httpd
ResourceManager [4886]: 2015/11/06 _ 11:53:14 info: Running/etc/init. d/httpd stop ### stop httpd service
ResourceManager [4886]: 2015/11/06 _ 11:53:14 info: Running/etc/ha. d/resource. d/IPaddr 192.168.21.1/24/eth0 stop
IPaddr [4975]: 2015/11/06 _ 11:53:14 INFO: ifconfig eth0: 0 down
IPaddr [4946]: 2015/11/06 _ 11:53:14 INFO: Success
Heartbeat [4873]: _ 11:53:14 info: All HA resources relinquished.
### Stop all HA resources on node 1 and kill heartbeat processes.
Heartbeat [4374]: 2015/11/06 _ 11:53:15 WARN: 1 lost packet (s) for [orasrv2.xlk.com] [95: 97]
Heartbeat [4374]: 2015/11/06 _ 11:53:15 info: No pkts missing from orasrv2.xlk.com!
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: killing HBREAD process 4381 with signal 15
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: killing HBFIFO process 4377 with signal 15
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: killing HBWRITE process 4378 with signal 15
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: killing HBREAD process 4379 with signal 15
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: killing HBWRITE process 4380 with signal 15
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: Core process 4378 exited. 5 remaining
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: Core process 4379 exited. 4 remaining
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: Core process 4381 exited. 3 remaining
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: Core process 4377 exited. 2 remaining
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: Core process 4380 exited. 1 remaining
Heartbeat [4374]: 2015/11/06 _ 11:53:16 info: orasrv1.xlk.com Heartbeat shutdown complete.

### Observe Node 2 and get the VIP at this time
[Root @ orasrv2 ~] # Ifconfig | grep eth-A1
Eth0 Link encap: Ethernet HWaddr 00: 0C: 29: F1: 2E: E4 # Author: Leshami
Inet addr: 192.168.21.13 Bcast: 192.168.21.255 Mask: 255.255.255.0
--
Eth0: 0 Link encap: Ethernet HWaddr 00: 0C: 29: F1: 2E: E4 # Blog: http://blog.csdn.net/leshami
Inet addr: 192.168.21.1 Bcast: 192.168.21.255 Mask: 255.255.255.0
--
Eth1 Link encap: Ethernet HWaddr 00: 0C: 29: F1: 2E: EE
Inet addr: 192.168.144.129 Bcast: 192.168.144.255 Mask: 255.255.255.0

### Verify the httpd service. In this case, access the home page through VIP and return data from orasrv2, that is, node 2.
Http: // 192.168.21.1/index.html
Orasrv2.xlk.com

4. Use hb_standby to switch
### Next, use hb_standby to switch the service back to node 1. Start heartbeat on node 1 before switching.
[Root @ orasrv2 ~] #/Usr/share/heartbeat/hb_standby
11:56:05 Going standby [all].

### Continue tracking node 1 logs
Heartbeat [5114]: 2015/11/06 _ 11:55:23 info: Version 2 support: false
Heartbeat [5114]: 2015/11/06 _ 11:55:23 WARN: Logging daemon is disabled -- enabling logging daemon is recommended
Heartbeat [5114]: _ 11:55:23 info :**************************
Heartbeat [5114]: 2015/11/06 _ 11:55:23 info: Configuration validated. Starting heartbeat 2.1.4
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: heartbeat: version 2.1.4
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: Heartbeat generation: 1446714159
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: glib: UDP Broadcast heartbeat started on port 694 (694) interface eth0
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: glib: UDP Broadcast heartbeat closed on port 694 interface eth0-Status: 1
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: glib: ping heartbeat started.
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: G_main_add_TriggerHandler: Added signal manual handler
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: G_main_add_TriggerHandler: Added signal manual handler
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: G_main_add_SignalHandler: Added signal handler for signal 17
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: Local status now set to: 'up'
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: Link 192.168.21.254: 192.168.21.254 up.
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: Status update for node 192.168.21.254: status ping
Heartbeat [5115]: 2015/11/06 _ 11:55:23 info: Link orasrv1.xlk.com: eth0 up.
Heartbeat [5115]: 2015/11/06 _ 11:55:25 info: Link orasrv2.xlk.com: eth0 up.
Heartbeat [5115]: 2015/11/06 _ 11:55:25 info: Status update for node orasrv2.xlk.com: status active
Harc [5124]: _ 11:55:25 info: Running/etc/ha. d/rc. d/status
Heartbeat [5115]: 2015/11/06 _ 11:55:25 info: Comm_now_up (): updating status to active
Heartbeat [5115]: 2015/11/06 _ 11:55:25 info: Local status now set to: 'active'
Heartbeat [5115]: 2015/11/06 _ 11:55:26 info: remote resource transition completed.
Heartbeat [5115]: 2015/11/06 _ 11:55:26 info: remote resource transition completed.
Heartbeat [5115]: 2015/11/06 _ 11:55:26 info: Local Resource acquisition completed. (none)
Heartbeat [5115]: 2015/11/06 _ 11:55:26 info: Initial resource acquisition complete (T_RESOURCES (them ))
### On the log surface, node 1 has been started and both nodes are active

### The following is the log information output after enabling hb_standby. All resources are taken over by node 1
Heartbeat [5115]: 2015/11/06 _ 11:56:06 info: orasrv2.xlk.com wants to go standby [all]
Heartbeat [5115]: 2015/11/06 _ 11:56:06 info: standby: acquire [all] resources from orasrv2.xlk.com
Heartbeat [5143]: 2015/11/06 _ 11:56:06 info: acquire all HA resources (standby ).
ResourceManager [5156]: 2015/11/06 _ 11:56:06 info: Acquiring resource group: orasrv1.xlk.com IPaddr: 192.168.21.1/24/eth0 httpd
IPaddr [5182]: 2015/11/06 _ 11:56:06 INFO: Resource is stopped
ResourceManager [5156]: 2015/11/06 _ 11:56:06 info: Running/etc/ha. d/resource. d/IPaddr 192.168.21.1/24/eth0 start
IPaddr [5277]: 2015/11/06 _ 11:56:06 INFO: Using calculated netmask for 192.168.21.1: 255.255.255.0
IPaddr [5277]: 2015/11/06 _ 11:56:06 INFO: eval ifconfig eth0: 0 192.168.21.1 netmask 255.255.255.0 broadcast 192.168.21.255
IPaddr [5248]: 2015/11/06 _ 11:56:06 INFO: Success
ResourceManager [5156]: 2015/11/06 _ 11:56:06 info: Running/etc/init. d/httpd start
Heartbeat [5143]: 2015/11/06 _ 11:56:07 info: all HA resource acquisition completed (standby ).
Heartbeat [5115]: 2015/11/06 _ 11:56:07 info: Standby resource acquisition done [all].
Heartbeat [5115]: 2015/11/06 _ 11:56:07 info: remote resource transition completed.

VII. Summary
A. switching between nodes in a cluster is essentially a resource takeover, which usually includes VIP allocation and recycling, disk unmounting and mounting, and service start/stop.
B. For slave nodes, the corresponding shared disk (non-strictly speaking) cannot be read/written, and the service is stopped.

Hot Standby for Web Services Based on Heartbeat V1

Heartbeat enables high-availability clusters of Web Services

Heartbeat + LVS + Ldirectord high-availability Load Balancing Solution

DRBD + Heartbeat + NFS High Availability Configuration notes

Heartbeat high availability for MySQL using NFS based on CRM

Heartbeat high-availability httpd simple configuration based on Resources

CentOS 6 High Availability service Heartbeat v3 installation and configuration

This article permanently updates the link address: