Recently encountered errors ORA-00600 [kjctr_pbmsg: badbmsg2], and lead to RAC node instance restart, the final confirmation of the problem caused by private network instability. ORA-00600: internalerrorcode, arguments: [kjctr_pb
Recently encountered errors ORA-00600 [kjctr_pbmsg: badbmsg2], and lead to RAC node instance restart, the final confirmation of the problem caused by private network instability. ORA-00600: internalerrorcode, arguments: [kjctr_pb
Recently encountered errors ORA-00600 [kjctr_pbmsg: badbmsg2], and lead to RAC node instance restart, the final confirmation of the problem caused by private network instability.
ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9FFFFFFFFC996B58], [0x9FFFFFFFFC9976B8], [], [], [], [], [], [] [], [], [] LMS1 (ospid: 12379): terminating the instance due to error 484
1. For detailed analysis, first view the log:
Alert log
Mon Aug 11 23:53:10 2014 Errors in file/oracle/app/oracle/diag/rdbms/cdrdb/orcl/trace/orcl_lms1_122.16.trc (incident = 1104178): ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9FFFFFFFFC996B58], [0x9ffffffc9976b8], [], [], [], [], [], [], [], [], [] Incident details in:/oracle/app/oracle/diag/rdbms/cdrdb/orcl/incident/incdir_1103698/w.aug 11 23:53:12 2014 dumdiagping nostic data in directory = [cdmp_201408425000012], requested by (instance = 1, osid = 12379 (LMS1), summary = [incident = 1104178]. use ADRCI or Support Workbench to package the incident. see Note 411.1 at My Oracle Support for error and packaging details. mon Aug 11 23:53:13 2014 Sweep [inc] [1104178]: completedSweep [inc2] [1104178]: completedErrors in file/oracle/app/oracle/diag/rdbms/cdrdb/orcl/trace/orcl_lms1_122.16.trc: ORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [comment], [0x9ffffffc9976b8], [], [], [], [], [], [], [], [], [], [] LMS1 (ospid: 12379 ): terminating the instance due to error 484Mon Aug 11 23:53:22 2014ORA-1092: opitsk aborting process
Orcl_lms%12%_i110%8.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0-64bit ProductionWith the Partitioning, Real Application Clusters, OLAP, data Miningand Real Application Testing optionsORACLE_HOME =/oracle/app/oracle/product/11.2.0/dbhome_1System name: HP-UXNode name: h7sd05daRelease: B .11.31Version: UMachine: ia64Instance name: orclRedo thread mounted by this instance: 1 Oracle process number: 14 Unix process pid: 12379, image: oracle1_sd05da (LMS1) Dump continued from file: /oracle/app/oracle/diag/rdbms/cdrdb/orcl/trace/orcl_lms1_12379.trcORA-00600: internal error code, arguments: [kjctr_pbmsg: badbmsg2], [0x9ffffffc996b58], [comment], [], [], [], [], [], [], [], [], [] ========= Dump for incident 1104178 (ORA 600 [kjctr_pbmsg: badbmsg2]) ======== *** 23:53:10. 339 dbkedDefDump (): Starting incident default dumps (flags = 0x2, level = 3, mask = 0x0) ----- SQL Statement (None) ----- Current SQL information unavailable-no cursor. ----- Call Stack Trace ----- skdstdst <-ksedst <-dbkedDefDump <-ksedmp <-ksfdmp <-$ handle <-dbgexProcessError <-dbgeExecuteForError <-dbgePostErrorKGE <-2352 <-ignore <-128 <-kgeadse <-kgerinv_internal <-kgerinv <-example <-kjctr_pbmsg <-example <-kjctrcv <-kjcsrmg <-kjmsm <-ksbrdp <-opirip <-opidrv <-sou2o <-opimai_real <-ssthrdmain <-main <-main_opd_entry --------------------- Binary Stack Dump ---------------------
2. Check the patch information. The current version is 11.2.0.2.1.
$ Opatch lsinventory Installed Top-level Products (1): Oracle Database 11g 11.2.0.2.0 Patch 10248523: applied on Fri Mar 25 09:33:02 GMT + 08:00 2011
3. Search for related documents and bugs based on the error and list the following bugs and descriptions.
Bug 18015296: ORA-600 [KJCTR_PBMSG: BADBMSG2] in 11.2.0.3
The assert is trigerred because the batch message is invalid/specified upt. This looks like some form of underlying infrastructure/network issue, Please work with customer to have this checked and tested.
Bug 18771858: LMS0 terminating the instance due to error 484 (ORA-00600 [KJCTR_PBMSG: BADBMSG2] in 11.2.0.3
From the past bug 16240464 & bug 18015296, both were closed by dev as not a product defect.
It was suggested that problem was outside Oracle stack at network level. so please check with CT on same lines to identify network problems (if any) with help from there OS/Net support. refer Doc ID 563566.1 Troubleshooting gc block lost and Poor Network Performance in a RAC Environment
Bug 16240464: instance crash with ORA-00600 [KJCTR_PBMSG: BADBMSG2] in 11.2.0.3
This looks like some form of underlying infrastructure/network issue, please work with customer to have this checked and tested.
Bug 17452853: LNX64-12.1-EF, db inst crash with LMS4 HIT ORA-600 [KJCTR_PBMSG: BADBMSG2] in 12.1.0.2
Bug 17049773 Diagnostic enhancement to give additional parameter in error ORA-600 [kjctr_pbmsg: badbmsg2] in 12.1.0.1
Note: This fix will not address the root cause of the error but the additional information may help with diagnosis of the cause.
Bug 13917456: LNX64-12.1-UD: asm lmd hit ORA-00600 KJCTR_PBMSG: BADBMSG2 IN NON-UPGRADED NODES in 12.1.0.0.2
It may occurred in upgrading stage from 11.2.0.3 to 12.1. Not related with this SR.
4. Now, I need to check the AWR, oswatcher and all the LMS, LMD, LMON, LMHB and DIAG logs when the problem occurs to see if there are multiple records.
At the same time, cluvfy and mongohk are used to check the overall environment of RAC.
--. AWR report ~ 23: 00 on Aug 11 from both nodes. --. deploy the oswatcher, then collect the current OS information, when the database workload is high. --. all the LMS, LMD, LMON, LMHB and DIAG from both nodes. --. CVU output: cluvfy stage-pre crsinst-n -Verbose --. Please run ‑heck as root. ‑hk-Health Checks for the Oracle Stack (Doc ID 1268927.2)
