Cluster Series 5 (high-availability cluster based on corosync + pacemaker)

Lab environment:
1) vmwere-based Virtual Machine implementation
2) This configuration has two Test Nodes, node1.a.com and node2.a.com, with the IP addresses 172.16.4.11 and 172.16.4.22 respectively;
3) The simulated cluster service is a web service;
4) the address for providing web services is 172.16.4.1.
Experiment topology:

1. Basic Environment Settings:
Preparations for configuring an HA Host:
1) because of the importance of cluster nodes and the need to provide a stable network environment, here we need to configure a fixed IP address for the node, edit/etc/sysconfig/network-scripts/ifcfg-eth0, the configuration IP address is as follows:
DEVICE = eth0
 
BOOTPROTO = static
 
IPADDR = 172.16.4.11
 
NETMASK = 255.255.0.0
 
ONBOOT = yes
 
HWADDR = 00: 0c: 29: a2: fa: bb

2) ensure that the name of the node host is consistent with the result of the "uname-n" command. edit/etc/sysconfig/network and configure it as follows:
NETWORKING = yes
 
NETWORKING_IPV6 = no
 
HOSTNAME = node1.a.com
Perform the following operations to make the host name take effect this time:
# Source/etc/sysconfig/network
# Hostname node1.a.com
3) The host name and corresponding IP Address Resolution Service of all nodes can work normally. We do not need the DNS service. We only need to ensure that the/etc/hosts files on the two nodes are the following content:

172.16.4.11 node1.a.com node1
 
172.16.4.22 node2.a.com node2
After completing the preceding three steps, you can resolve the host names as follows:

4) configure the node trust relationship:
Node 1:
# Ssh-keygen-t rsa
# Ssh-copy-id-I ~ /. Ssh/id_rsa.pub root @ node2
Node 2:
# Ssh-keygen-t rsa
# Ssh-copy-id-I ~ /. Ssh/id_rsa.pub root @ node1
Then test:

2. corresponding software packages required for the installation environment:
# Yum install-y libibverbs librdmacm lm_sensors libtool-ltdl openhpi-libs openhpi perl-TimeDate
3. Install corosync and pacemaker. Put the software package in the/root/corosync directory (both nodes are required ):
Cluster-glue-1.0.6-1.6.el5.i386.rpm
 
Cluster-glue-libs-1.0.6-1.6.el5.i386.rpm
 
Corosync-1.2.7-1.1.el5.i386.rpm
 
Corosynclib-1.2.7-1.1.el5.i386.rpm
 
Heartbeat-3.0.3-2.3.el5.i386.rpm
 
Heartbeat-libs-3.0.3-2.3.el5.i386.rpm
 
Libesmtp-1.0.4-5.el5.i386.rpm
 
Openais-1.1.3-1.6.el5.i386.rpm
 
Openaislib-1.1.3-1.6.el5.i386.rpm
 
Pacemaker-1.0.11-1.2.el5.i386.rpm
 
Pacemaker-libs-1.0.11-1.2.el5.i386.rpm
 
Perl-TimeDate-1.16-5.el5.noarch.rpm
 
Resource-agents-1.0.4-1.1.el5.i386.rpm
The software version you installed may be different from the author's, as long as it meets your experimental platform.
Start installation:
# Cd/root/corosync/
# Yum-y -- nogpgcheck localinstall *. rpm
Here we use the local yum installation and ignore the packet check.
4. corosync configuration (executed on node 1 ):
# Cd/etc/corosync
# Cp corosync. conf. example corosync. conf
Corosync. conf. example is the configuration sample. You only need to copy it and modify it:
# Vim/etc/corosync. conf
 
# Please read the corosync. conf.5 manual page
 
Compatibility: whitetank
 
Totem {
 
Version: 2
 
Secauth: off
 
Threads: 0
 
Interface {
 
Ringnumber: 0
 
Bindnetaddr: 172.16.0.0 ------ Here is the network address of the network card you want to modify
 
Mcastaddr: 226.94.1.1
 
Mcastport: 5405
 
}
 
}
 
Logging {
 
Fileline: off
 
To_stderr: no
 
To_logfile: yes
 
To_syslog: yes
 
Logfile:/var/log/cluster/corosync. log ----- Where the log is stored
 
Debug: off
 
Timestamp: on
 
Logger_subsys {
 
Subsys: AMF
 
Debug: off
 
}
 
}
 
Amf {
 
Mode: disabled
 
}
 
Service {----------------- here the following content needs to be added
 
Ver: 0
 
Name: pacemaker
 
}
 
Aisexec {
 
User: root
 
Group: root
 
}
Generate the authentication key file used for inter-node communication:
# Corosync-keygen --- An authentication key file is generated in the current directory.
Then copy the relevant file to node 2:
# Scp-p corosync. conf authkey node2:/etc/corosync/
The directory where the logs generated by corosync are created for the two nodes:
# Mkdir/var/log/cluster
# Ssh node2 -- mkdir/var/log/cluster
5. Start corosync (run on node 1 ):
#/Etc/init. d/corosync start
Starting Corosync Cluster Engine (corosync): [OK] "------" indicates that your corosync has been started.
Start to verify if there are other errors:
Check whether the corosync engine is properly started
# Grep-e "Corosync Cluster Engine"-e "configuration file"/var/log/messages
 
Sep 15 11:20:41 localhost smartd [2350]: Opened configuration file/etc/smartd. conf
 
Sep 15 11:24:24 localhost smartd [2416]: Opened configuration file/etc/smartd. conf
 
Sep 15 11:30:14 localhost smartd [2659]: Opened configuration file/etc/smartd. conf
 
Sep 15 15:12:38 localhost corosync [694]: [MAIN] Corosync Cluster Engine ('1. 2.7 '): started and ready to provide service.
 
Sep 15 15:12:38 localhost corosync [694]: [MAIN] Successfully read main configuration file '/etc/corosync. conf '.
Check whether the initialization member node notification is normal
# Grep TOTEM/var/log/messages
 
Sep 15 15:12:38 localhost corosync [694]: [TOTEM] Initializing transport (UDP/IP ).
 
Sep 15 15:12:38 localhost corosync [694]: [TOTEM] Initializing transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0 ).
 
Sep 15 15:12:38 localhost corosync [694]: [TOTEM] The network interface [172.16.4.11] is now up.
 
Sep 15 15:12:39 localhost corosync [694]: [TOTEM] Process pause detected for 515 MS, flushing membership messages.
 
Sep 15 15:12:39 localhost corosync [694]: [TOTEM] A processor joined or left the membership and a new membership was formed.

Check whether errors occur during startup.
# Grep ERROR:/var/log/messages | grep-v unpack_resources
If no error occurs, no information is displayed here.
Check if pacemaker is started properly
# Grep pcmk_startup/var/log/messages
Sep 15 15:12:39 localhost corosync [694]: [pcmk] info: pcmk_startup: CRM: Initialized
 
Sep 15 15:12:39 localhost corosync [694]: [pcmk] Logging: Initialized pcmk_startup
 
Sep 15 15:12:39 localhost corosync [694]: [pcmk] info: pcmk_startup: Maximum core file size is: 4294967295
 
Sep 15 15:12:39 localhost corosync [694]: [pcmk] info: pcmk_startup: Service: 9
 
Sep 15 15:12:39 localhost corosync [694]: [pcmk] info: pcmk_startup: Local hostname: node1.a.com

If the above verification fails, you can start Node 2:
# Ssh node2 --/etc/init. d/corosync start ----- You Need To perform this step on node 1
Starting Corosync Cluster Engine (corosync): [OK] ---- This indicates that your node 2corosync has been started. You must continue to verify whether an exception error has occurred on node 2, perform the verification error step.
View the running status of a node:
# Crm status
 
================
 
Last updated: Thu Sep 15 15:17:50 2011
 
Stack: openais
 
Current DC: node1.a.com-partition with quorum
 
Version: 1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87
 
2 Nodes configured, 2 expected votes
 
0 Resources configured.
 
================
 
Online: [node1.a.com node2.a.com] ---- This indicates that both of your cluster nodes are running normally.
6. Configure the working attributes of the cluster:
Because corosync has enabled stonith by default, but no stonith device is added, the default configuration is unavailable:
# Crm_verify-L
 
Crm_verify [762]: 2011/09/15 _ 15:20:13 ERROR: unpack_resources: Resource start-up disabled since no STONITH resources have been defined
 
Crm_verify [762]: 2011/09/15 _ 15:20:13 ERROR: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option
 
Crm_verify [762]: 2011/09/15 _ 15:20:13 ERROR: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity
 
Errors found during check: config not valid
 
-V may provide more details
To prevent future errors and affect operations, we can disable stonith here:
# Crm configure property stonith-enabled = false ---- this command will be submitted and will take effect immediately
INFO: building help index
Run the following command to view the current corosync Configuration:
# Crm configure show
 
Node node1.a.com
 
Node node2.a.com
 
Property $ id = "cib-bootstrap-options "\
 
Dc-version = "1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87 "\
 
Cluster-infrastructure = "openais "\
 
Expected-quorum-votes = "2 "\
 
Stonith-enabled = "false" '---- stonith has been disabled
The commands related to crm and crm_verify are the command line-based cluster management tools provided by pacemaker Versions later than 1.0. They can be executed on any node in the cluster to view the relevant information.
 
7. Add cluster resources to the cluster:
Corosync supports resource proxies such as heartbeat, lsb, and ocf. Currently, the most common types are lsb and lsb. The stonith class is used to configure stonith devices.
View the resource proxy types supported by the current cluster system:
# Crm ra classes
 
Heartbeat
 
Lsb
 
Ocf/heartbeat pacemaker
 
Stonith
To view the list of resource proxies:
# Crm ra list lsb

# Crm ra list ocf heartbeat

# Crm ra list ocf pacemaker

# Crm ra list stonith
 

The following is a simple example of implementing a web Cluster:
Install httpd and configure the corresponding test webpage:
# Yum-y install httpd
# Echo "# Chkconfig httpd off
# Service httpd stop
# Ssh node2 -- 'echo "# Ssh node2 -- 'chkconfig httpd off'
# Ssh node2 -- 'service httpd stop'
Add web resources:
First, create an IP address resource for the created web cluster,
# Crm configure primitive WebIP ocf: heartbeat: IPaddr params ip = 172.16.4.1
Then add the httpd service as a cluster resource. Adding httpd as a cluster resource has two resource proxies available: lsb and ocf: heartbeat. For simplicity, we use the lsb type here:
# Crm configure primitive WebSite lsb: httpd
In this case, you can enter http: // 172.16.4.1 in the browser of the host to access the web Service:

Ps: In this experiment, I first added web Server resources before installing the httpd service. In the end, an error occurred while viewing the node information. The test page cannot be accessed. solution: restart the corosync service. Of course, this is not allowed in actual production.
View node status information:
# Crm status
================
 
Last updated: Thu Sep 15 15:51:17 2011
 
Stack: openais
 
Current DC: node1.a.com-partition with quorum
 
Version: 1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87
 
2 Nodes configured, 2 expected votes
 
2 Resources configured.
 
================
 
Online: [node1.a.com node2.a.com]
 
WebIP (ocf: heartbeat: IPaddr): Started node1.a.com
 
WebSite (lsb: httpd): Started node2.a.com
We found that IP resources and web site resources are located on two nodes respectively. In actual production, our website resources are also located on different nodes; we can also divide the two resources into one group for only one site:
# Crm configure group Web WebIP WebSite
# Crm status
 
================
 
Last updated: Thu Sep 15 15:54:31 2011
 
Stack: openais
 
Current DC: node1.a.com-partition with quorum
 
Version: 1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87
 
2 Nodes configured, 2 expected votes
 
1 Resources configured.
 
================
 
Online: [node1.a.com node2.a.com]
 
Resource Group: Web
 
WebIP (ocf: heartbeat: IPaddr): Started node1.a.com
 
WebSite (lsb: httpd): Started node1.a.com ----- note that it has changed to the same node
Subsequent tests:
1. Make Node 1 offline on node 2:
# Ssh node1 --/etc/init. d/corosync stop
# Crm status
================
 
Last updated: Thu Sep 15 15:57:31 2011
 
Stack: openais
 
Current DC: node2.a.com-partition WITHOUT quorum
 
Version: 1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87
 
2 Nodes configured, 2 expected votes
 
1 Resources configured.
 
================
 
Online: [node2.a.com]
 
OFFLINE: [node1.a.com]
At this time, node 1 is offline, but Node 2 cannot obtain the corresponding resources and of course cannot access the corresponding services. This is not the result we want, in this case, the cluster status is "WITHOUT quorum". WITHOUT quorum, the node cannot obtain the corresponding resources and the cluster service cannot run properly. We can set quorum to ignore quorum. The following settings:
# Crm configure property no-quorum-policy = ignore
# Crm status
================
 
Last updated: Thu Sep 15 16:00:24 2011
 
Stack: openais
 
Current DC: node2.a.com-partition WITHOUT quorum
 
Version: 1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87
 
2 Nodes configured, 2 expected votes
 
1 Resources configured.
 
================
 
Online: [node2.a.com]
 
OFFLINE: [node1.a.com]
 
Resource Group: Web
 
WebIP (ocf: heartbeat: IPaddr): Started node2.a.com
 
WebSite (lsb: httpd): Started node2.a.com "-------" Node 2 has obtained resources.
The test webpage is as follows:

If you start corosync of Node 1 again, what would happen to you?
# Ssh node1 --/etc/init. d/corosync start
# Crm status
================
 
Last updated: Thu Sep 15 16:02:09 2011
 
Stack: openais
 
Current DC: node2.a.com-partition with quorum
 
Version: 1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87
 
2 Nodes configured, 2 expected votes
 
1 Resources configured.
 
================
 
Online: [node1.a.com node2.a.com]
 
Resource Group: Web
 
WebIP (ocf: heartbeat: IPaddr): Started node1.a.com
 
WebSite (lsb: httpd): Started node1.a.com ------ the resource is retrieved by node 1 again
The test webpage is as follows:

After Node 1 is started, resources are re-obtained. If the device performance of Node 1 is very good and important, we can allow it to happen, but if the node performance is equal, and the two nodes are not very close to each other. This kind of resource is constantly "easy to master", which will make it unable to be accessed during that time. Therefore, sometimes, after resources fail to be transferred to other nodes due to node failure, resources are prohibited from being transferred back even if the original node returns to normal. This can be achieved by defining the resource stickiness (stickiness. You can specify resource stickiness when creating a resource or after creating a resource.
 
Range of resource stickiness values and their functions:
Equal to 0: This is the default option. Resources are placed in the most suitable position in the system, which means that resources are transferred only when the load capacity is "good" or the node with poor load becomes available. This option is basically equivalent to automatic fault recovery, but resources may be transferred to non-active nodes;
Greater than 0: the resource is more willing to stay in the current position, but it will move when a more suitable node is available. The higher the value, the more willing the resource to stay in the current position;
Less than 0: the resource is more willing to move from the current location. The higher the absolute value, the more willing the resource to leave the current location;
INFINITY: If the resource is forcibly transferred because the node is not suitable for running resources (node shutdown, node standby, reaching migration-threshold, or configuration change), the resource is always in the current position. This option serves almost the same purpose as disabling automatic failover;
-INFINITY: resources are always moved from the current location;
 
The author's plain understanding is: You have no money, and anyone is willing to contact you. the richer you are, the more I will be reluctant to give up on you!
 
Manually set the default viscosity value of the resource:
# Crm configure rsc_defaults resource-stickiness = 100
 
 
Here we have another concept: resource constraints
From the above experiments, we know that the resources WebIP and WebSite may run on two nodes respectively. This is not true for applications that provide Web services through this IP address, both resources must run on a node at the same time.
It can be seen that, even if the cluster has all the necessary resources, it may not be able to handle them correctly. Resource constraints are used to specify the cluster nodes on which resources are running, the sequence in which resources are loaded, and the other resources on which specific resources are dependent. Pacemaker provides three resource constraints:
1) Resource Location: defines the nodes on which resources can, cannot, or can be run as much as possible;
2) Resource Collocation (Resource arrangement): scheduling constraints are used to define whether cluster resources can or cannot run on a node simultaneously;
3) Resource Order: Order constraints define the Order in which cluster resources are started on nodes;

You also need to specify the score when defining the constraint. Scores are an important part of the cluster's working methods. In fact, the entire process from migrating resources to deciding which resources to stop in the degraded cluster is achieved by modifying the score in some way. Scores are calculated based on each resource. No node with a negative resource score can run the resource. After calculating the resource score, the cluster selects the node with the highest score. INFINITY is currently defined as 1,000,000. Addition or subtraction of infinity follows the following three basic rules:
1) any value + infinity = infinity
2) any value-infinity =-infinity
3) Infinity-infinity =-infinity

When defining resource constraints, you can also specify the scores of each constraint. The score indicates the value assigned to this resource constraint. The constraint with a higher score is applied first, and the constraint with a lower score is then applied. By using different scores to create more location constraints for a given resource, you can specify the order of the target node for resource failover.

The preceding WebIP and WebSite may run on different nodes. We can also solve the problem through the following methods:
# Crm configure colocation website-with-ip INFINITY: WebSite WebIP

We can also set the resource startup sequence. The WebIP runs first and then runs WebSite:
# Crm configure order httpd-after-ip mandatory: WebIP WebSite

In addition, in some cases, we may want the service to always run on a node with high performance normally. This can be achieved through location constraints:
# Crm configure location prefer-node1 WebSite rule 200: node1
This command restricts the WebSite to node1 and sets the score to 200;
 
 

Author: "Long marks"