Five most common problems that cause RAC instance crashes

Five most common problems that cause RAC instance crashes

Five most common problems that cause the RAC instance to crash (Document ID 1549191.1)

Applicable:

OracleDatabase-Enterprise Edition-version 11.2.0.1 and later
The information contained in this document applies to all platforms

Purpose

This document aims to summarize the five most common problems that may cause the RAC instance to crash and the common problems reported by earlier versions (such as 10.2.0.5.

Applicability

Issue 1 to 5 only applies to 11gR2 RAC. The <version> issue only applies to the mentioned version.

Details

 

Problem 1: ORA-29770 LMHB termination instance

 

Symptoms:

LMON (ospid: 31216) waits for event 'control file sequential read' for 88 secs.
Errors in file/oracle/base/diag/rdbms/prod/prod3/trace/prod3_lmhb_31304.trc (incident = 2329 ):
ORA-29770: global enqueue process LMON (OSID 31216) is hung for more than 70 seconds
LMHB (ospid: 31304) is terminating the instance.

Or
LMON (ospid: 8594) waits for event 'control file sequential read' for 118 secs.
ERROR: LMON is not healthy and has no heartbeat.
ERROR: LMHB (ospid: 8614) is terminating the instance.

Possible causes:

LMON waits to read the control file, causing LMHB to crash the instance
Bug 11890804 LMHB crashes instance withORA-29770 after long "control file sequential read" waits

Solution:

Bug 8888434 has been fixed in 11.2.0.2 and later versions.
Bug 11890804 has been fixed in 11.2.0.3 and later versions.
See Document 1197674.1, Document 8888434.8, and Document 11890804.8 for details.

 

Problem 2: instance crash caused by ORA-481

Symptoms:

1. PMON (ospid: 12585): terminating the instance due to error 481
The LMON process trace file is displayed as follows:
Begin DRM (1, 107) (swin 0)
* Drm quiesce <kjxgmrcfg: Reconfiguration started, type 6

LMS <x> process trace file display:
10:53:44. 218905: Start affinity expansion for pkey 81885.0
10:53:44. 498923: Expand failed: pkey 81885.0, 229 shadowstraversed, 153 replayed 1 retries

2. PMON (ospid: 4915562): terminating the instance due to error 481
Sat Oct 01 19:21:37 2011
System state dump requested by (instance = 2, osid = 4915562 (PMON), summary = [abnormal instance termination].

Possible causes:

1. Bug 11875294 LMS gets stuck during DRM, Instance crashed with ORA-481
2. HAIP is offline on some cluster nodes, or HAIP is online on all cluster nodes, but cannot communicate through it. For example, the ping operation fails.

Solution:

1. Bug 11875294 has been fixed in 11.2.0.3. The method to bypass the problem is:
Set
_ Gc_read_mostly_locking = FALSE to disable read mostly.
For more information, see <Document 11875294.8>.

2. To fix HAIP problems, see Document 1383737.1

Problem 3: ORA-600 [kjbmp rlst: shadow], ORA-600 [kjbrref: pkey], ORA-600 [kjbmocvt: rid], [kjbclose_remaster :! Drm], ORA-600 [kjbrasr: pkey] causes instance crash

Symptoms:

Since ORA-600 [kjbmp rlst: shadow], ORA-600 [kjbrref: pkey], ORA-600 [kjbmocvt: rid], [kjbclose_remaster :! Drm] Or ORA-600 [kjbrasr: pkey] causes RAC instance crash

Possible causes:

This set of ORA-600 is related to either the DRM (dynamic resourceremastering) message or the read mostly lock. Multiple bugs are involved, including:
Document 9458781.8 Missing close message tomaster leaves closed lock dangling crashing the instance with assorted Internalerror
Document 9835264.8 ORA-600 [kjbrasr: pkey]/ORA-600 [kjbmocvt: rid] in RAC with dynamic remastering
Document 10200390.8 ORA-600 [kjbclose_remaster :! Drm] in RAC with fix for 9979039
Document 10121589.8 ORA-600 [kjbmp rlst: shadow] can occur in RAC
Document 11785390.8 Stack rollback uption/incorrect behaviour possible in RAC
Document 12408350.8 ORA-600 [kjbrasr: pkey] in RAC with read mostly locking
Document 12834027.8 ORA-600 [kjbmp rlst: shadow]/ORA-600 [kjbrasr: pkey] with RAC read mostly locking

Solution:

Most of the above bugs are corrected in 11.2.0.3. Installing the 11.2.0.3 Patch Set should avoid these bugs. Except for bug 12834027, this bug will be corrected in 12.1. The method to bypass this bug is:

Disable DRM
Or
Disable read mostly
For example, set "_ gc_read_mostly_locking" = FALSE

For instructions and solutions for each bug, see the above documentation.

 

Problem 4: kcldle/kclfplz/kcbbxsv_l2/kclfprm is generated after flash cache is enabled, leading to instance crash

Symptoms:

The ORA-7445 [kcldle] is reported in the Alert Log
ORA-7445 [kclfplz]
ORA-7445 [kcbbxsv_12]
ORA-744 [kclfprm]

Possible causes:

They are caused by different bugs, and these bugs all come down to the basic bug 12337941 Dumps on kcldle/kclfplz/kcbbxsv_l2/kclfprm using flash

Solution:

This bug has been fixed in 11.2.0.3. Install the patch set or use the following method to bypass this issue: Disable Flash Cache
For more information, see Document 12337941.8.

 

Problem 5: LMS reports ORA-600 [kclpdc_21] error, instance crash

Symptoms:

The ORA-600 [kclpdc_21] is reported in the Alert Log

Possible causes:

Document 10040035.8 LMS gets ORA-600 [kclpdc_21] and instance crashes

Solution:

This bug has been fixed in 11.2.0.3.

10.2.0.5 Problems

Symptoms:

1. LMS process reports ORA-600 [kjccgmb: 1] error causes instance crash, LMS <n>: terminating instance due to error 484
2. The instance crashes due to the following reasons:
Received an instance abort message from instance 2 (reason 0x0)
Please check instance 2 alert and LMON trace files for detail.
LMD0: terminating instance due to error 481

Possible causes:

1. Bug 11893577-lmd crashed with ORA-00600 [KJCCGMB: 1]
2. Bug 9577274-1OFF: unable to view request output and log after applying fixto issue in bug 9400041

Solution:

1. For 10.2.0.5.0, install the merged patch 12616787
2. Install the merged patch 13470618 for 10.2.0.5.5.
At the time of writing this article, only specific platforms have available patches. For any version 10.2.0.5.x, you do not need to install the above two patches at the same time.

Install Oracle 11gR2 (x64) in CentOS 6.4)

Steps for installing Oracle 11gR2 in vmwarevm

Install Oracle 11g XE R2 In Debian