ORA-00600 [kjctr_pbmsg: badbmsg2]

ORA-00600 [kjctr_pbmsg: badbmsg2]

Recently encountered errors ORA-00600 [kjctr_pbmsg: badbmsg2], and lead to RAC node instance restart, the final confirmation of the problem caused by private network instability.

ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9FFFFFFFFC996B58], [0x9FFFFFFFFC9976B8], [], [], [], [], [], [] [], [], []

LMS1 (ospid: 12379): terminating the instance due to error 484

1. For detailed analysis, first view the log:
Alert log

Mon Aug 11 23:53:10 2014

Errors in file/Oracle/app/oracle/diag/rdbms/cdrdb/orcl/trace/orcl_lms1_122.16.trc (incident = 1104178 ):

ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9FFFFFFFFC996B58], [0x9FFFFFFFFC9976B8], [], [], [], [], [], [] [], [], []

Incident details in:/oracle/app/oracle/diag/rdbms/cdrdb/orcl/incident/incdir_11020.8/orcl_lms1_121__i1101_8.trc

Mon Aug 11 23:53:12 2014

Dumping diagnostic data in directory = [cdmp_2014081_1_12], requested by (instance = 1, osid = 12379 (LMS1), summary = [incident = 1104178].

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

Mon Aug 11 23:53:13 2014

Sweep [inc] [1104178]: completed

Sweep [inc2] [1104178]: completed

Errors in file/oracle/app/oracle/diag/rdbms/cdrdb/orcl/trace/orcl_lms1_122.16.trc:

ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9FFFFFFFFC996B58], [0x9FFFFFFFFC9976B8], [], [], [], [], [], [] [], [], []

LMS1 (ospid: 12379): terminating the instance due to error 484

Mon Aug 11 23:53:22 2014

ORA-1092: opitsk aborting process

Orcl_lms%12%_i110%8.trc

Oracle Database 11g Enterprise Edition Release 11.2.0.2.0-64bit Production

With the Partitioning, Real Application Clusters, OLAP, Data Mining

And Real Application Testing options

ORACLE_HOME =/oracle/app/oracle/product/11.2.0/dbhome_1

System name: HP-UX

Node name: mongosd05da

Release: B .11.31

Version: U

Machine: ia64

Instance name: orcl

Redo thread mounted by this instance: 1

Oracle process number: 14

Unix process pid: 12379, image: oracle%sd05da (LMS1)

Dump continued from file:/oracle/app/oracle/diag/rdbms/cdrdb/orcl/trace/orcl_lms1_122.16.trc

ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9FFFFFFFFC996B58], [0x9FFFFFFFFC9976B8], [], [], [], [], [], [] [], [], []

========= Dump for incident 1104178 (ORA 600 [kjctr_pbmsg: badbmsg2]) ========

* ** 23:53:10. 339

DbkedDefDump (): Starting incident default dumps (flags = 0x2, level = 3, mask = 0x0)

----- SQL Statement (None )-----

Current SQL information unavailable-no cursor.

----- Call Stack Trace -----

Skdstdst <-ksedst <-dbkedDefDump <-ksedmp <-ksfdmp

<-$ Cold_dbgexPhaseII <-dbgexProcessError <-dbgeExecuteForError <-dbgePostErrorKGE <-2352

<-DbkePostKGE_kgsf <-128 <-kgeadse <-kgerinv_internal <-kgerinv

<-Kgeasnmierr <-kjctr_pbmsg <-kjctr_rksxp <-kjctrcv <-kjcsrmg

<-Kjmsm <-ksbrdp <-opirip <-opidrv <-sou2o

<-Opimai_real <-ssthrdmain <-main <-main_opd_entry

--------------------- Binary Stack Dump ---------------------

2. Check the patch information. The current version is 11.2.0.2.1.

$ Opatch lsinventory

Installed Top-level Products (1 ):

Oracle Database 11g 11.2.0.2.0

Patch 10248523: applied on Fri Mar 25 09:33:02 GMT + 2011

3. Search for related documents and bugs based on the error and list the following bugs and descriptions.

Bug 18015296: ORA-600 [KJCTR_PBMSG: BADBMSG2] in 11.2.0.3
The assert is trigerred because the batch message is invalid/specified upt. This looks like some form of underlying infrastructure/network issue, Please work with customer to have this checked and tested.
Bug 18771858: LMS0 terminating the instance due to error 484 (ORA-00600 [KJCTR_PBMSG: BADBMSG2] in 11.2.0.3
From the past bug 16240464 & bug 18015296, both were closed by dev as not a product defect.
It was suggested that problem was outside Oracle stack at network level. so please check with CT on same lines to identify network problems (if any) with help from there OS/Net support. refer Doc ID 563566.1 Troubleshooting gc block lost and Poor Network Performance in a RAC Environment
Bug 16240464: instance crash with ORA-00600 [KJCTR_PBMSG: BADBMSG2] in 11.2.0.3
This looks like some form of underlying infrastructure/network issue, please work with customer to have this checked and tested.
Bug 17452853: LNX64-12.1-EF, db inst crash with LMS4 HIT ORA-600 [KJCTR_PBMSG: BADBMSG2] in 12.1.0.2
Bug 17049773 Diagnostic enhancement to give additional parameter in error ORA-600 [kjctr_pbmsg: badbmsg2] in 12.1.0.1
Note: This fix will not address the root cause of the error but the additional information may help with diagnosis of the cause.
Bug 13917456: LNX64-12.1-UD: asm lmd hit ORA-00600 KJCTR_PBMSG: BADBMSG2 IN NON-UPGRADED NODES in 12.1.0.0.2
It may occurred in upgrading stage from 11.2.0.3 to 12.1. Not related with this SR.

4. Now, I need to check the AWR, oswatcher and all the LMS, LMD, LMON, LMHB and DIAG logs when the problem occurs to see if there are multiple records.
At the same time, cluvfy and mongohk are used to check the overall environment of RAC.

--. AWR report ~ 23: 00 on Aug 11 from both nodes.

--. Deploy the oswatcher, then collect the current OS information, when the database workload is high.

-- All the LMS, LMD, LMON, LMHB and DIAG from both nodes.

--. CVU output:

Cluvfy stage-pre crsinst-n <node1, node2>-verbose

--. Please run has heck as root.

Managing HK-Health Checks for the Oracle Stack (Doc ID 1268927.2)

5. when checking AWR, we found "gc blocks lost". Theoretically, if the private network is normal, this error will not occur. The appearance of this error can basically be explained, the private network is unstable.

Awrrpt_2_29557_29558.html

Snap Id Snap Time Sessions Cursors/Session

Begin Snap: 29557 11-Aug-14 22:00:45 563 1.3

End Snap: 29558 11-Aug-14 23:01:00 551 1.3

Elapsed: 60.24 (mins)

DB Time: 4,835.90 (mins)

Top 5 Timed Foreground Events

Event Waits Time (s) Avg wait (MS) % DB time Wait Class

Db file sequential read 6,269,185 185,621 30 63.97 User I/O

Database CPU 42,433 14.62

Gc current grant 2-way 3,251,636 25,671 8 8.85 Cluster

Db file scattered read 550,524 9,873 18 3.40 User I/O

Gc cr multi block request 637,442 6,790 11 2.34 Cluster

Instance Activity Stats

Statistic Total per Second per Trans

Gc blocks lost 269 0.07 0.01 <

Awrrpt_1_29557_29558.html

Snap Id Snap Time Sessions Cursors/Session

Begin Snap: 29557 11-Aug-14 22:00:44 2470 1.0

End Snap: 29558 11-Aug-14 23:00:59 2500 1.0

Elapsed: 60.25 (mins)

DB Time: 4,549.47 (mins)

Top 5 Timed Foreground Events

Event Waits Time (s) Avg wait (MS) % DB time Wait Class

Db file sequential read 8,180,795 154,504 19 56.60 User I/O

Database CPU 44,994 16.48

Gc current grant 2-way 3,699,003 29,357 8 10.75 Cluster

Db file scattered read 677,065 10,190 15 3.73 User I/O

Gc cr multi block request 718,327 7,856 11 2.88 Cluster

Statistic Total per Second per Trans

Gc blocks lost 410 0.11 0.01 <

6. This error proves the possibility of private network problems. The final conclusion is as follows:

The Bugs 16240464 and 18015296 are raised for the similar issue and both the bugs are closed as "Vendor OS Problem ".
The bug confirmed that this issue is cause because of logical block upload uption during network transfer over the interconnect or Infrastructure issue.

The ORA-00600 [kjctr_pbmsg: badbmsg2] error is purely a result of unstable network.
From the AWR reports it is confirmed that we were seeing block lost during the problematic time frame. This is one of the evidurthat network is either saturated or causing packets to be upted.

By the way, Checked the AWR report. Found "gc blocks lost ".
Please involve the OS team and Network team to identify the root cause of the issue. The below note will helpful for the network issue.
Troubleshooting gc block lost and Poor Network Performance in a RAC Environment (Doc ID 563566.1)

7. there is still no more powerful evidence to deal with this problem, that is, the oswatcher log. If there is a problem with the oswatcher log, it will make the private network issue clearer, after all, the "gc blocks lost" and ORA-00600 [kjctr_pbmsg: badbmsg2] errors encountered during the entire problem analysis process are reported from the oracle database perspective and cannot convince the OS engineers, if the oswatcher log records TCP and UDP packet loss at that time, the problem will be clearer and the responsibility will be clearer.

For how to install and use oswatcher, see OSWatcher (Doc ID 301137.1)

-------------------------------------- Split line --------------------------------------

Oracle 11g installation manual on RedHat Linux 5.8 _ x64 Platform

Installing Oracle 12C in Linux-6-64

Install Oracle 11gR2 (x64) in CentOS 6.4)

Steps for installing Oracle 11gR2 in vmwarevm

Install Oracle 11g XE R2 In Debian

-------------------------------------- Split line --------------------------------------