Oracle cluster heartbeat and Its Parameter misscount/disktimeout/reboottime, oracle Cluster

Oracle cluster heartbeat and Its Parameter misscount/disktimeout/reboottime, oracle Cluster

In Oracle RAC, you can check the health status of RAC from multiple levels and different mechanisms, that is, you can isolate faults through the heartbeat mechanism and certain voting algorithms. If a node fails, the faulty node will be evicted from the cluster to prevent the faulty node from damaging data. This article describes several heartbeat mechanisms and the adjustment of Heartbeat parameters in Oracle RAC.

 

I. OCSSD and CSS
OCSSD is a Linux or Unix process that manages and provides Cluster Synchronization Services (CSS) Services. The Oracle user runs the process and provides the node member management function. Once the process fails, the node restarts. The CSS Service provides two heartbeat mechanisms: Network heartbeat and disk heartbeat. Both have the maximum latency. The network heartbeat latency is MC (Misscount), and the disk heartbeat latency is IOT (I/O Timeout ). The two parameters are in seconds. If the parameter is missing, Misscount <Disktimeout is used. The two heartbeat mechanisms are described below.

 

Ii. Network heartbeat
Therefore, the node status is detected through the private network. If the hardware and software of the private network make the private network between cluster nodes unable to communicate normally within a certain period of time, this leads to split-brain. Because the storage in the cluster environment is shared storage, the faulty nodes must be isolated from the cluster to avoid data disasters. The specific action of this network heartbeat is described as follows:
Every one second, a sending thread in the cssd sends a network tcp heartbeat to itself and all nodes. The processing thread of the ocssd. bin has es the heartbeat.
If the package network is dropped or has error, the error correction mechanic on tcp wocould retransmit the package.
Oracle does not retransmit. from the ocssd. log, you will see a WARNING message about missing of heartbeat if a node does not receive a heartbeat from another node for 15 seconds (50% of miscount ). another warning is reported in ocssd. log if the same node is missing for 22 seconds (75% of miscount ).. another warning continues from the same node for 27 seconds (90% miscount ). when the heartbeat is missing 100% .. 30 seconds miscount, the node is evicted
 
The Network heartbeat latency is called misscount and can be queried and modified using the crsctl tool.
[Grid @ Linux-01 ~] $ Crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
 
The above query results show that if the inline network latency between nodes in the cluster is greater than 30 s, Oracle considers that there is a split brain between nodes and the faulty node needs to be evicted from the cluster.
Oracle uses the voting algorithm to determine how to find faulty nodes. The following is an example of an algorithm description. For more information, see Oracle RAC.
Each node in the cluster needs a heartbeat mechanism to notify each other of the "Health Status". If each node receives a "notification", it means one vote. For clusters with three nodes, each node has three tickets during normal operation. When the heartbeat of node A fails but node A is still running, the entire cluster is split into two small partitions. Node A is one, and the remaining two are one. This requires removing a partition to ensure the healthy operation of the cluster. For clusters with these three nodes, after A heartbeat problem occurs, B and C are A partion with two votes and A has only one vote. According to the voting algorithm, clusters composed of B and C obtain control, and A is removed. If there are only two nodes, the voting algorithm becomes invalid. Because each node has only one vote. In this case, you need to introduce the third Device: Quorum Device. Quorum Device usually uses a shared disk, which is also called Quorum disk. This Quorum Disk also represents one ticket. When the heartbeat of the two nodes fails, the two nodes fight for the Quorum Disk ticket at the same time, and the first request to arrive will be satisfied first. Therefore, the first node to obtain Quorum Disk will receive two tickets. The other node will be removed.
 
After a node is isolated, it is usually restarted before 11gR2. In 11gR2, ClusterWare first attempts to shut down all resources of the node and cleans up the failed components in the cluster, that is, restarting the failed components. If the failed component is not successfully cleared, restart the node to force the cleaning.

 

Iii. Disk heartbeat
A thread in ocssd. bin updates the voting disk every second.
If a node does not update the voting disks for 200 seconds, it's evicted.
However, the ocssd. bin on the local node has the logic that it will bring down the node if it has an I/O error more than majority of the voting disks. also there is a CRS reconfiguration is happening when misscount is 27 second and the local node is rebooted. as a result, you rarely see an eviction due to failure of the voting disk on 10.2.0.4 (this is more common in 10.2.0.1) because the ocssd. bin will abort the node before it get evicted by another node if writing to the voting disk is the problem.
As mentioned above, each node updates the voting disk every second. A shared voting disk is used to check the disk heartbeat. If the ocssd process updates the voting disk for more than 200 s, that is, the disktimeout value, Oracle considers the voting disk to be offline, and generates a voting disk offline record in the Clusterware alarm log. If the number of offline voting disks on the current node is less than the number of online voting disks, the node will survive. If the number of offline voting disks is greater than or equal to the number of online voting disks, clusterware considers that the disk heartbeat is faulty and the faulty node is evicted from the cluster to perform automatic repair. For example, if three voting disks exist and A voting disk on node A appears offline, clusterware generates offline records in the alarm log for offline disks (1) <online disks (2, however, no action is taken. If two or more voting disks are offline on the current node, the offline disk (2)> online disk (1) will be kicked out of the cluster.

 

Iv. RebootTime Parameters
Note that this RebootTime parameter is also important. The default value is 3 s.
Default 3 seconds-the amount of time allowed for a node to complete a reboot
After the CSS daemon has been evicted.
Crsctl get css reboottime
# Author: Leshami
# Blog: http://blog.csdn.net/leshami

 

5. Adjustment of Heartbeat Parameters
1) 10.2.0.2 to 11.1.0.7
A) Shut down CRS on all but one node. For exact steps use note 309542.1
B) Execute crsctl as root to modify the misscount:
$ CRS_HOME/bin/crsctl set css misscount <n >#### where <n> is the maximum private network latency in seconds
$ CRS_HOME/bin/crsctl set css reboottime <r> [-force] #### (<r> is seconds)
$ CRS_HOME/bin/crsctl set css disktimeout <d> [-force] #### (<d> is seconds)
C) Reboot the node where adjustment was made
D) Start all other nodes which was shutdown in step 1
E) Execute crsctl as root to confirm the change:
$ CRS_HOME/bin/crsctl get css misscount
$ CRS_HOME/bin/crsctl get css reboottime
$ CRS_HOME/bin/crsctl get css disktimeout

 

2) how to modify 11gR2
With 11gR2, these settings can be changed online without taking any node down:

A) Execute crsctl as root to modify the misscount:
$ CRS_HOME/bin/crsctl set css misscount <n >#### where <n> is the maximum private network latency in seconds
$ CRS_HOME/bin/crsctl set css reboottime <r> [-force] #### (<r> is seconds)
$ CRS_HOME/bin/crsctl set css disktimeout <d> [-force] #### (<d> is seconds)
B) Execute crsctl as root to confirm the change:
$ CRS_HOME/bin/crsctl get css misscount
$ CRS_HOME/bin/crsctl get css reboottime
$ CRS_HOME/bin/crsctl get css disktimeout

 

What is the split-brain split in the ORACLE database RAC cluster?

Oracle rac css provides two background services: Group Management (GM) and Node Monitor (NM), where the GM management group (Group) and lock (lock) services. At any time in the cluster, a node will always act as the GM master node ). Other nodes in the cluster send GM requests to the master node serially, and the master node broadcasts the change information of the cluster members to other nodes in the cluster. Group membership is synchronized every time a cluster reconfiguration is reset. Each node independently interprets the changes of cluster members.
The node monitoring NM Service maintains node information consistency with cluster software of other vendors through skgxn (skgxn-libskgxn.a, database that provides node monitoring. In addition, NM also provides maintenance for the well-known Network heartbeat and Disk heartbeat to ensure that the node remains alive. When a cluster member does not have a normal Network heartbeat or Disk heartbeat, NM is responsible for kicking the member out of the Cluster. If the Member is kicked out of the cluster, the node will be restarted (reboot ).
The NM service uses records in OCR (Interconnect information is recorded in OCR) to understand the endpoints for listening and interaction, and sends heartbeat information over the network to other cluster members. At the same time, it also monitors Network heartbeat from all other cluster members. This heartbeat happens every second. If the Network heartbeat of a node is at misscount (by the way: in 10.2.0.1, the default value of misscount is 60 s for Linux, and that of other platforms is 30 s. If a third-party vendor clusterware is used, the value of disktimeout is 600 s, but disktimeout is not introduced in 10.2.0.1; If 10.2.0.4 is later, disktimeout is 200 s; misscount is 30 s after 11.2: CRS-4678: Successful get misscount 30 for Cluster Synchronization Services, CRS-4678: Successful get disktimeout 200 for Cluster Synchronization Services) Specified If the node is not received in seconds, the node is considered dead. NM is also responsible for initializing the cluster reset (Initiates cluster reconfiguration) when other nodes join or leave the cluster ).
In scenarios where split-brain occurs, NM also monitors voting disks to learn about other competing sub-clusters (subclusters ). It is necessary to introduce the sub-clusters. Imagine that there are a large number of nodes in our environment. It is our imagination to use the environment of the 128 nodes officially built by Oracle, when a network failure occurs, there are multiple possibilities. One possibility is that the global network fails, that is, each of the 128 nodes cannot have a network heartbeat, at this time, up to 128 "isolated island" sub-clusters are generated. Another possibility is local network failure. The 128 nodes are divided into multiple parts, each of which contains more than one node, these parts can be called subclusters ). When a network failure occurs, multiple nodes in the sub-cluster can still communicate with each other to transmit the voting information (vote mesg). However, the sub-clusters or isolated island nodes cannot communicate with each other through the conventional Interconnect Network, in this case, the voting disk is required for the NM Reconfiguration.
Because NM uses voting disk to solve communication barriers caused by network faults, it is necessary to ensure that voti... the remaining full text>