Linux Enterprise Cluster notes-ch9 stonith and ipfail

1. this chapter introduces stonith in detail. When both servers attempt to take over resources, split-brain may occur. When the heartbeat line is faulty, this situation is most likely to occur. This situation is fatal. It may cause the service to fail to run normally. Worse, it may cause data destruction because both servers may read and write a data source at this time, this can cause problems. Even if both servers write only their own storage devices, the storage content of the two servers may be inconsistent. Therefore, we should avoid this situation as much as possible, the solution provided by heartbeat is-stonith

2. stonith must be able to run normally and require certain hardware devices. This device has the following features: it can be connected to two servers of HA (or connected to one server) and can send commands to the device through software commands, the device then directly restarts or loses power on the controlled server. In this way, a server is restarted or shut down quickly to avoid split-brain.

3. This section describes an example. This example is not very common and has some limitations, because only one stonith device is used in this example. The device connects to a server. In other words, if the stonith device is connected to the primary server, in this example, the backup server can only send commands to restart the primary server. The primary server cannot restart or shut down the backup server.

4. OK. The figure of the hardware connection in this example is as follows: attachment 1

5. as shown in the attachment diagram, this example has the following limitations: (1) all resources can only run on the primary server (2) failover can only run from the primary server-> Backup Server, because once a Failover occurs, the backup server sends a command to shut down or restart the primary server to prevent split-brain.

6. OK. In this architecture, if a Failover occurs, it will look like this. See Appendix 2.

7. it is clear that when heartbeat disappears (it may be because the primary server is down or the heartbeat line is faulty), the backup server sends the reset command through the stonith device, thus, the primary server is forcibly restarted. If auto_failback is configured as on, the control of the resource may be retrieved again after the primary server is restarted. However, in this way, this effectively avoids split-brain.

8. Then, a list of some stonith devices currently supported by heartbeat is provided: See appendix 1.

9. In addition to the real devices above, heartbeat also supports some fake devices. Through these devices, we can simulate the situation to debug whether stonith can work normally. See Appendix 2.

10. After heartbeat's stonith package is installed, you can use this command to view the list of stonith devices supported by heartbeat:/usr/sbin/stonith-l

11. The following describes the meatware device. This is a fake stonith device, but it can be used to debug stonith for us. After a meatware stonith device is configured, stonith will not send any command to stonith device, then we will simply publish an operator alert (this information can be viewed in the heartbeat log). In this way, let's test stonith.

12. Run the following command to create a meatware device:

#/Usr/sbin/stonith-T meatware-P "" chilly

This command creates a meatware type stonith device for host chilly, and parameter is empty.

After executing this command, view the/var/log/messages file and you will find:

Stonith: operator intervention required to reset test.
Stonith: Run "meatclient-C test" after power-loading ing the machine.

This indicates that stonith asked us to run the meatclient-C chilly command to perform a test on restarting the chilly machine. If we have a chilly machine, we need to manually restart the machine and run the meatclient-C chilly command to clear the stonith event. Since we do not have the chilly machine, execute meatclient-C chilly directly to clear this event. If you run this command or, the following error occurs:

Warning!
If Server "chilly" has not been manually power-cycled or disconnected from all shared resources and networks, data on shared disks may become upted and migrated services might not work as expected.
Please verify that the name or address abve corresponds to the server you just rebooted.
Proceed? [YN]

After pressing y, the following error occurs:

Meatware_client: reset confirmed.

13. in this way, a meatware device is created, and then we can go to the HA. configured in CF, in the/etc/ha of the two servers. d/ha. add the following line in Cf:

Stonith_host * meatware

OK. The second parameter is used to tell heartbeat which server is connected to the stonith device. *, it indicates that both servers are physically connected to the stonith device. stonith can perform reset operations on any server, because we use meatware, so of course we can write it like this, if it is a real device, write it as needed.

14. in this way, you can start the heartbeat of both servers, and then kill-9 to kill the heartbeat process of the primary server (do not use commands such as service heartbeat stop, this command uninstalls the resources on the primary server safely, and the backup server receives the commands smoothly, which cannot create a split-brain scenario. Only forced killing can be performed to simulate Server failure scenarios ), observe the/var/log/messages of the backup server:

Backupserver heartbeat [835]: info :**************************
Backupserver heartbeat [835]: info: configuration validated. Starting heartbeat <version>
Backupserver heartbeat [836]: info: Heartbeat: version <version>
Backupserver heartbeat [836]: info: Heartbeat generation: 3
Backupserver heartbeat [836]: info: UDP broadcast heartbeat started on port 694 (694) interface eth0
Backupserver heartbeat [841]: info: status update for server backupserver: status up
Backupserver heartbeat: info: running/etc/ha. d/rc. d/status
Backupserver heartbeat [841]: info: link backupserver: eth0 up.
Backupserver heartbeat [841]: Warn: Server primaryserver: is dead
Backupserver heartbeat [841]: info: status update for server backupserver: Status active
Backupserver heartbeat [847]: info: resetting server primaryserver with [meatware stonith device]
Backupserver heartbeat [847]: operator intervention required to reset primaryserver.
Backupserver heartbeat [847]: Run "meatclient-C primaryserver" after power-loading ing the machine.
Backupserver heartbeat: info: running/usr/local/etc/ha. d/rc. d/status
Backupserver heartbeat [852]: info: no local resources [/usr/local/lib/heartbeat/ResourceManager listkeys backupserver]
Backupserver heartbeat [852]: info: resource acquisition completed.

As shown in the preceding log, the backup server does not immediately start the sendmail service (Sendmail is the resource in this example), but waits for us to clear the meatware stonith event, this means that we need to manually restart the primary server, and then use commands such as meatclient-C to clear the event, so that heartbeat will think OK, the primary server restarts, backup Server can safely start Sendmail ,:)

15. Okay, then we will start clearing this event:

Backupserver> meatclient-C primaryserver

In this case, the backup server log is as follows:

Backupserver heartbeat [847]: Server primaryserver Meatware-reset.backupserver heartbeat [847]: info: Server primaryserver now reset.
Backupserver heartbeat [841]: info: resources being acquired from primaryserver.
Backupserver heartbeat: info: running/usr/local/etc/ha. d/rc. d/stonith
Backupserver heartbeat: info: running/usr/local/etc/ha. d/rc. d/status
Backupserver heartbeat: stonith complete
Backupserver heartbeat: info: Taking Over Resource Group sendmail
Backupserver heartbeat: info: acquiring Resource Group: primaryserver sendmail
Backupserver heartbeat: info: running/etc/init. d/sendmail start

OK. the backup server starts Sendmail.

16. Use a real stonith device. Obviously, each stonith device has different configurations. Execute the command stonith-help to see how to configure these devices. The syntax for configuring a real stonith device is as follows:

Stonith: Config File Syntax: <serial_device> <Server> <outlet> [<Server> <outlet> [...]

For example:

Stonith_host backupserver rps10/dev/ttys0 primaryserver.mydomain.com 0

This configuration indicates that the backup server can control the primary server through stonith device, and the stonith device connects to the backup server through/dev/ttys0 (string line ), primary server is connected to outlet 0 (plug-in) of stonith device named rps10.

It is still relatively simple. Note that many stonith devices are connected to the server through a network cable. At this time, you may need to configure the username and password for Logon (enhancing security, prevent hackers from restarting these servers easily. In addition, it is also possible to directly integrate HP and stonith devices into the server in many large factories.

17. avoid Multiple stonith events. If we configure auto_failback to on and stonith restarts the server instead of shutting down the server, a loop of stonith may occur, such as primary server down, stonith device restarts him. When the primary server gets up and takes over the resources and goes down again, repeat the above process. This is the stonith many times. It is very easy to solve this problem, directly power off primary server, instead of reset it. There are two ways to do this. First, modify the source code of heartbeat (exaggerated ):

Rc = s-> s_ops-> reset_req (S, st_generic_reset, nodename );

Modify:

Rc = s-> s_ops-> reset_req (S, st_poweroff, nodename );

Or the second method is to modify the BIOS settings of the primary server. In the BIOS of many machines, you can set the reset signal to be sent to execute the shutdown action instead of the reset.

17. network failures. So far, we have made a lot of efforts to avoid spof, but it is not enough, such as the network failure mentioned in this section. This is the case. We all know that servers provide external services over the network. What if there is a problem with this service network? (The heartbeat line is correct.) in this case, clients cannot communicate with the primary server over the network, so the service is suspended. There are two solutions for this problem:

(1) run a third-party monitoring program to monitor whether the primary server can normally access the network. If a communication exception is found, the monitoring program will shut down the heartbeat daemon on the primary server. In this case, failover occurs, the backup server takes over the service.
(2) Using the ipfail plug-in provided by heartbeat, We can configure ipfail in heartbeat to regularly ping a batch of servers, ipfail will ask the backup server "is your network normal? ", If the backup server can access the network normally, failover occurs.

This section describes the second method:

Respawn hacluster/usr/lib/heartbeat/ipfail
Ping 10.1.1.254 10.1.1.253
Auto_failback off

From the above we can see that we started the ipfail daemon in the name of the hacluster user, and then pinged the 10.1.1.254 and 10.1.1.253 servers. These configurations can be added to the last part of the/etc/ha. d/ha. cf file (before the server configuration)

18. watchdog and softdog. watchdog is divided into two types: hardware and software. The hardware is a dedicated hardware dog. When an OS exception is detected, the machine will be restarted. The software dog is a softdog daemon that comes with Linux kernel, if the kernel is not supported, it is necessary to re-compile the kernel. Generally, the RedHat feature kernel is available, which is a module.

19. this section only describes softdog. You can use modprobe to mount the softdog module. After mounting the module, when a system exception is detected, softdog places the system in the state of a kernel panic, however, we want to restart the system rather than the panic system, which can be solved through this Configuration:

(1) modify the startup parameters of LILO or grub and add: append = "panic = 60" before image ="

(2) This method is simpler, as follows: # echo 60>/proc/sys/kernel/panic. The default panic time is 0. After it is changed to 60, the system will be restarted.

20. Heartbeat also supports watchdog. Add the following to the/etc/ha. d/ha. cf file:

Watchdog/dev/Watchdog

In this way, the dog will monitor the heartbeat process. If we kill the heartbeat process, the system will restart (because the software dog is working)

21. Heartbeat has been fully explained, and LVS will be introduced later. Here is a summary of how to test the heartbeat configuration. Some suggestions are provided in the book. The content outline is listed here. For details, refer to the following content:

(1) unplug the power cord on the primary server
(2) Test the behavior of the hb_standby command
(3) unplug the production network cable on the primary server
(4) remove one of the heartbeat paths between the two servers
(5) Remove all of the heartbeat paths between the two servers
(6) Kill the heartbeat daemon on the primary server
(7) Kill the resource daemons on the primary server (if we use cl_respawn, heartbeat will automatically restart the resource, when the status script of the resource script does not return OK)
(8) reboot both servers