Oracle Database ORA-600 k2vcbk_2 Failure recovery

A friend came to me and said they couldn't start the database, the database started Ora-600[k2vcbk_2] Error, the database version is 11.2.0.2 RAC, the operating system is AIX 6.1
sql> Recover database;
Media recovery complete.
sql> ALTER DATABASE open;
ALTER DATABASE Open
*
ERROR at line 1:
Ora-01092:oracle instance terminated. Disconnection forced
Ora-00600:internal error code, arguments: [K2vcbk_2], [], [], [], [], [], [],
[], [], [], [], []
Process id:7930020
Session id:49 Serial number:14761
Database node, 1st log
Mon Sep 21 15:45:41 2015
Thread 1 advanced to log sequence 54076 (LGWR switch)
Current log# seq# 54076 mem# 0: +dg01/xifenfei/onlinelog/group_13.332.779459035
Current log# seq# 54076 mem# 1: +dg01/xifenfei/onlinelog/group_13.344.779582621
Mon Sep 21 15:45:44 2015
Archived Log Entry 74655 added for thread 1 sequence 54075 ID 0x5a0bc0e1 dest 1:
Mon Sep 21 15:56:18 2015
Errors in File/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_18088342.trc (incident=184348):
ORA-00600: Internal error code, parameters: [Kturpots_0], [], [], [], [], [], [], [], [], [], [], []
Incident details in:/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_184348/xifenfei1_ora_18088342_ I184348.trc
Mon Sep 21 15:56:34 2015
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
Error trapped in 2PC on transaction 7.16.120119. Cleaning up.
Error Stack returned to User:
ORA-00600: Internal error code, parameters: [Kturpots_0], [], [], [], [], [], [], [], [], [], [], []
Errors in File/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_ora_18088342.trc (incident=184349):
Ora-00603:oracle server session terminated due to fatal error
ORA-00600: Internal error code, parameters: [Kturpots_0], [], [], [], [], [], [], [], [], [], [], []
Mon Sep 21 15:56:34 2015
Dumping diagnostic data in directory=[cdmp_20150921155634], requested by (Instance=1, osid=18088342), summary=[incident =184348].
Incident details in:/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_184349/xifenfei1_ora_18088342_ I184349.trc
Mon Sep 21 15:56:35 2015
Sweep [inc][184349]: Completed
Sweep [inc][184348]: Completed
Sweep [inc2][184348]: Completed
OPIODR aborting process unknown ospid (18088342) as a result of ORA-603
Mon Sep 21 15:57:12 2015
Errors in File/oracle/diag/rdbms/xifenfei/xifenfei1/trace/xifenfei1_smon_7536810.trc (incident=184274):
Ora-00600:internal error code, arguments: [K2vcbk_2], [], [], [], [], [], [], [], [], [], [], []
Incident details in:/oracle/diag/rdbms/xifenfei/xifenfei1/incident/incdir_184274/xifenfei1_smon_7536810_ I184274.trc
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
Mon Sep 21 15:57:16 2015
Dumping diagnostic data in directory=[cdmp_20150921155716], requested by (Instance=1, osid=7536810 (SMON)), summary=[ INCIDENT=184274].
Fatal Internal error happened while Smon is doing active transaction recovery.
Errors in FILE/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI1/TRACE/XIFENFEI1_SMON_7536810.TRC:
Ora-00600:internal error code, arguments: [K2vcbk_2], [], [], [], [], [], [], [], [], [], [], []
Smon (ospid:7536810): Terminating the instance due to error 474
Mon Sep 21 15:57:18 2015
Ora-1092:opitsk aborting Process
Database node, 2nd log
Mon Sep 21 15:21:50 2015
Archived Log Entry 74653 added for thread 2 sequence 23559 ID 0x5a0bc0e1 dest 1:
Mon Sep 21 15:44:28 2015
Thread 2 advanced to log sequence 23561 (LGWR switch)
Current log# seq# 23561 mem# 0: +dg01/xifenfei/onlinelog/group_12.338.779457003
Current log# seq# 23561 mem# 1: +dg01/xifenfei/onlinelog/group_12.265.779582493
Mon Sep 21 15:44:31 2015
Archived Log Entry 74654 added for thread 2 sequence 23560 ID 0x5a0bc0e1 dest 1:
Mon Sep 21 15:45:31 2015
Distrib TRAN xifenfei.1ebab0a5.20.3.1533822
Is local tran 20.3.1533822 (hex=14.03.17677e)
Insert pending committed Tran, scn=14590688068086 (HEX=D45.28C781F6)
Mon Sep 21 15:45:31 2015
Distrib TRAN xifenfei.1ebab0a5.20.3.1533822
Is local tran 20.3.1533822 (hex=14.03.17677e))
Delete pending committed Tran, scn=14590688068086 (HEX=D45.28C781F6)
Mon Sep 21 15:56:35 2015
Dumping diagnostic data in directory=[cdmp_20150921155634], requested by (Instance=1, osid=18088342), summary=[incident =184348].
Mon Sep 21 15:57:10 2015
Error 3135 trapped in 2PC on transaction 20.11.1534704. Cleaning up.
Error Stack returned to User:
ORA-03135: Connection lost contact
OPIDCL aborting process unknown ospid (9175532) as a result of ORA-604
Mon Sep 21 15:57:17 2015
Dumping diagnostic data in directory=[cdmp_20150921155716], requested by (Instance=1, osid=7536810 (SMON)), summary=[ INCIDENT=184274].
Mon Sep 21 15:57:23 2015
Reconfiguration started (Old Inc, New Inc 20)
List of instances:
2 (Myinst:2)
Global Resource Directory Frozen
* Dead instance detected-domain 0 Invalid = TRUE
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Mon Sep 21 15:57:23 2015
LMS 2:3 GCS Shadows cancelled, 1 closed, 0 Xw survived
Mon Sep 21 15:57:23 2015
LMS 0:2 GCS Shadows cancelled, 0 closed, 0 Xw survived
Mon Sep 21 15:57:23 2015
LMS 1:3 GCS Shadows cancelled, 1 closed, 0 Xw survived
Set Master Node Info
Submitted all Remote-enqueue requests
Dwn-cvts replayed, Valblks dubious
All grantable Enqueues granted
Post Smon to start 1st pass IR
Mon Sep 21 15:57:23 2015
Minact-scn:inst 2 is now the master inc#:20 Mmon proc-id:6816208 status:0x7
MINACT-SCN status:grec-scn:0x0000.00000000 gmin-scn:0x0d45.28c2bb5c gcalc-scn:0x0d45.28c3bd2e
Minact-scn:master found Reconf/inst-rec before RECSCN scan old-inc#:20 new-inc#:20
Mon Sep 21 15:57:23 2015
Instance recovery:looking for dead threads
Submitted all GCS Remote-cache requests
Post Smon to start 1st pass IR
Fix write in GCs
Reconfiguration complete
Beginning instance recovery of 1 threads
Parallel recovery started with processes
Started Redo scan
Completed Redo Scan
Read 12626 KB redo, 1724 data blocks need recovery
Started Redo application at
Thread 1:logseq 54076, Block 184416
Recovery of Online Redo log:thread 1 Group Seq 54076 Reading Mem 0
mem# 0: +dg01/xifenfei/onlinelog/group_13.332.779459035
mem# 1: +dg01/xifenfei/onlinelog/group_13.344.779582621
Completed Redo Application of 9.78MB
Completed instance Recovery at
Thread 1:logseq 54076, Block 209669, SCN 14590688357285
1633 data blocks read, 1794 data blocks written, 12626 redo K-bytes Read
Thread 1 advanced to log sequence 54077 (thread recovery)
Mon Sep 21 15:57:33 2015
Error 3113 trapped in 2PC on transaction 21.18.1965522. Cleaning up.
Redo thread 1 internally disabled at SEQ 54077 (Smon)
Error Stack returned to User:
ORA-02050: Transaction 21.18.1965522 has been rolled back, some remote databases may have problems
ORA-03113: End of file for communication channel
ORA-02063: Immediately following line (from Zsk)
Mon Sep 21 15:57:34 2015
Archived Log Entry 74656 added for thread 1 sequence 54076 ID 0x5a0bc0e1 dest 1:
Mon Sep 21 15:57:34 2015
arc0:archiving disabled thread 1 sequence 54077
Archived Log Entry 74657 added for thread 1 sequence 54077 ID 0x5a0bc0e1 dest 1:
Mon Sep 21 15:57:35 2015
Thread 2 advanced to log sequence 23562 (LGWR switch)
Current log# 8 seq# 23562 mem# 0: +dg01/xifenfei/onlinelog/group_8.334.779456945
Current log# 8 seq# 23562 mem# 1: +dg01/xifenfei/onlinelog/group_8.267.779582453
Mon Sep 21 15:57:36 2015
Errors in File/oracle/diag/rdbms/xifenfei/xifenfei2/trace/xifenfei2_smon_6750672.trc (incident=200218):
Ora-00600:internal error code, arguments: [K2vcbk_2], [], [], [], [], [], [], [], [], [], [], []
Incident details in:/oracle/diag/rdbms/xifenfei/xifenfei2/incident/incdir_200218/xifenfei2_smon_6750672_ I200218.trc
Archived Log Entry 74658 added for thread 2 sequence 23561 ID 0x5a0bc0e1 dest 1:
Mon Sep 21 15:57:38 2015
Minact-scn:master continuing after IR
Mon Sep 21 15:57:41 2015
Dumping diagnostic data in directory=[cdmp_20150921155741], requested by (instance=2, osid=6750672 (SMON)), summary=[ INCIDENT=200218].
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
Fatal Internal error happened while Smon is doing instance transaction.
Errors in FILE/ORACLE/DIAG/RDBMS/XIFENFEI/XIFENFEI2/TRACE/XIFENFEI2_SMON_6750672.TRC:
Ora-00600:internal error code, arguments: [K2vcbk_2], [], [], [], [], [], [], [], [], [], [], []
Smon (ospid:6750672): Terminating the instance due to error 474
Mon Sep 21 15:57:41 2015
Ora-1092:opitsk aborting Process
Mon Sep 21 15:57:42 2015
Ora-1092:opitsk aborting Process
Mon Sep 21 15:57:42 2015
License High Water mark = 291
Instance terminated by Smon, PID = 6750672
USER (ospid:18874814): terminating the instance
Through the database log can be seen, because of Node 2 distributed transaction exception, and in 11.2.0.2, distributed transactions across nodes, causing node 2 Pmon cleanup exception transactions, but because of bugs, so that the exception transaction can not be cleaned up, resulting in nodes 1 crash, node 1 After Crash Node 2 is restored, and because of a distributed transaction bug, Smon rollback fails and the instance is crash. Unable to rollback causes the database to not start correctly, by querying MOS discovery to locate is bug 10222544 ORA-600 [k2vpci_2] from Multi-Branch Distributed transaction

For this type of problem, because the distributed transaction can not be cleaned, the processing method is to find out the transaction manual or directly shielded the transaction to solve the problem