[Translated from mos] What is split-brain in Oracle Clusterware and RAC? mosclusterware

[Translated from mos] What is split-brain in Oracle Clusterware and RAC? mosclusterware

What is split-brain in Oracle Clusterware and RAC?

Source:
What is Split Brain in Oracle Clusterware and Real Application Cluster (Document ID 1425586.1)

Applicable:
Oracle Database-Enterprise Edition-Version 10.1.0.2 and later
Information in this document applies to any platform.

Purpose:
This article explains split-brain in Oracle Clusterware and RAC, as well as errors and results related to split-brain.

Details:
In general terms, split-brain indicates data inconsistency. This Database Inconsistency originated from the overlap of two different datasets in the range.
Either because of the network design between servers or a faulty environment, the environment is based on mutual communication and unified data between servers.

Two components may experience split-brain:

1. Clusterware layer:
Cluster nodes maintain their heartbeats through the private network and voting disk.
When a private network is damaged and the time period specified by misscount setting occurs, the cluster nodes cannot communicate with each other through the private network, and split-brain occurs.
In this case, voting disk is used to determine which node survived and which node is evict out of the cluster. The general voting result is as follows:

 a.The group with more cluster nodes survive b.The group with lower node member in case of same number of node(s) available in each group c.Some improvement has been made to ensure node(s) with lower load survive in case the eviction is caused by high system load.

Generally, when split-brain occurs, the following information is displayed in ocssd. log:

[ CSSD]2011-01-12 23:23:08.090 [1262557536] >TRACE: clssnmCheckDskInfo: Checking disk info...[ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: clssnmCheckDskInfo: Aborting local node to avoid splitbrain.[ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: : my node(2), Leader(2), Size(1) VS Node(1), Leader(1), Size(2)[ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: ###################################[ CSSD]2011-01-12 23:23:08.090 [1262557536] >ERROR: clssscExit: CSSD aborting###################################

The above information shows that the communication from node 2 to node 1 does not work, So node 2 can only see one node (that is, node 2 itself), but node 1 works normally, node 1 can see two nodes in the cluster. To avoid split-brain, node 2 aborted itself.

Solution: contact the network administrator to check the private network layer to eliminate any network problems.

2. RAC (database) layer
To ensure data consistency, each instance in the RAC Database must maintain heartbeat with other instances. Heartbeat is maintained by the background processes LMON, LMD, LMS, and LCK.
Any process in these processes experiences IPC Send time out, which will lead to communication reconfiguration and instance eviction to avoid split-brain.
Similar to the voting disk on the clusterware layer, the control file is used to determine which instance survived and which instance is evict.
The voting result is similar to clusterware voting result. As the result, 1 or more instance (s) will be evicted.

Common messages in instance alert log are similar:

alert log of instance 1:---------Mon Dec 07 19:43:05 2011IPC Send timeout detected.Sender: ospid 26318Receiver: inst 2 binc 554466600 ospid 29940IPC Send timeout to 2.0 inc 8 for msg type 65521 from opid 20Mon Dec 07 19:43:07 2011Communications reconfiguration: instance_number 2Mon Dec 07 19:43:07 2011Trace dumping is performing id=[cdmp_20091207194307]Waiting for clusterware split-brain resolutionMon Dec 07 19:53:07 2011Evicting instance 2 from clusterWaiting for instances to leave: 2 ...alert log of instance 2:---------Mon Dec 07 19:42:18 2011IPC Send timeout detected. Receiver ospid 29940Mon Dec 07 19:42:18 2011Errors in file /u01/app/oracle/diag/rdbms/bd/BD2/trace/BD2_lmd0_29940.trc:Trace dumping is performing id=[cdmp_20091207194307]Mon Dec 07 19:42:20 2011Waiting for clusterware split-brain resolutionMon Dec 07 19:44:45 2011ERROR: LMS0 (ospid: 29942) detects an idle connection to instance 1Mon Dec 07 19:44:51 2011ERROR: LMD0 (ospid: 29940) detects an idle connection to instance 1Mon Dec 07 19:45:38 2011ERROR: LMS1 (ospid: 29954) detects an idle connection to instance 1Mon Dec 07 19:52:27 2011Errors in file /u01/app/oracle/diag/rdbms/bd/BD2/trace/PVBD2_lmon_29938.trc  (incident=90153):ORA-29740: evicted by member 0, group incarnation 10Incident details in: /u01/app/oracle/diag/rdbms/bd/BD2/incident/incdir_90153/BD2_lmon_29938_i90153.trc

In the preceding example, instance 2 LMD0 (pid 29940) is the proceser in IPC Send timeout. There cocould be various reasons causing IPC Send timeout. For example:

A. Network problem
B. Process hang
C. Bug etc

Please see Top 5 issues for Instance Eviction Document 1374110.1 for more information.

In the case of instance eviction, alert log and all background traces need to be checked to determine the root cause.

Known Issues1. Bug 7653579 - IPC send timeout in RAC after only short period Document 7653579.8    Refer: ORA-29740 Instance (ASM/DB) eviction on Solaris SPARC Document 761717.1    Fixed in: 11.2.0.1, 11.1.0.7.2 PSU and 11.1.0.7 Patch 22 on Windows2. Unpublished Bug 8267580: Wrong Instance Evicted Under High CPU Load    Refer: Wrong Instance Evicted Under High CPU Load in 11.1.0.7 Document 1373749.1    Fixed in: 11.2.0.13. Bug 8365141 - DRM quiesce step hang causes instance eviction Document 8365141.8    Fixed in: 10.2.0.5, 11.1.0.7.3, 11.1.0.7 patch 25 for Windows and 11.2.0.14. Bug 7587008 - Hung RAC instance not evicted from cluster Document  7587008.8    Fixed in: 10.2.0.4.4, 10.2.0.5 and 11.2.0.1, one-off patch available for various 11.1.0.7 release5. Bug 11890804 - LMHB crashes instance with ORA-29770 after long "control file sequential read" waits Document 11890804.8    Fixed in 11.2.0.2.5, 11.2.0.3 and 11.2.0.2 Patch 10 on Windows6. BUG:13732226 - NODE GETS EVICTED WITH REASON CODE 0X2    BUG:13399435 - KJFCDRMRCFG WAITED 249 SECS FOR LMD TO RECEIVE ALL FTDONES, REQUESTING KILL    BUG:13503204 - INSTANCE EVICTION DUE TO REASON 0X200000    Refer: 11gR2: LMON received an instance eviction notification from instance n Document 1440892.1    Fixed in: 11.2.0.4 and some merge patch available for 11.2.0.2 and 11.2.0.3