ORA-600 (qersqCloseRem-2) Error

ORA-600 (qersqCloseRem-2) error the customer's 10.2.0.4 RAC for Hp-un Environment encountered this error. Error message: Wed Feb 29 19:42:05 2012 Errors in file/opt/app/oracle/admin/orcl/udump/orcl1_ora_11261.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] www.2cto.com ORA-02068: following severe error from WEBDB.COMORA-03113: end-of-file on communication channelWed Feb 29 19:42:05 2012 Errors in file/opt/app/oracle/admin/orcl/udump/orcl1_ora_32036.tr C: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [], [] ORA-02068: following severe error from WEBDB.COMORA-03113: end-of-file on communication channelWed Feb 29 19:42:05 2012 Errors in file/opt/app/oracle/admin/orcl/udump/orclwing ora_5935.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [], [] ORA-0206 8: following severe error from WEBDB.COMORA-03113: end-of-file on communication channelWed Feb 29 19:42:05 2012 www.2cto.com Errors in file/opt/app/oracle/admin/orcl/udump/orcl1_ora_5026.trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [], [] ORA-02068: following severe error from WEBDB.COMORA-03113: end-of-file on communication channelWed Feb 29 19:42:05 2012 Errors in file/opt/app/oracle/admin/orcl/udump/orclloud ora_761_trc: ORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] ORA-02068: following severe error from WEBDB.COMORA-03113: end-of-file on communication channelWed Feb 29 19:42:08 2012 Trace dumping is refreshing Ming id = [cdmp_20120229194207] Wed Feb 29 19:42:17 2012 Trace dumping is perfor Ming id = [cdmp_20120229194217] This ORA-600 [qersqCloseRem-2] error is very rare, there is no recording on MOS. However, the tb error message is further analyzed. This error occurs when access to the remote database is abnormal. Check for further details: *** 19:42:05. 564 ksedmp: internal or fatal errorORA-00600: internal error code, arguments: [qersqCloseRem-2], [Invalid Handle], [], [], [], [], [], [] www.2cto.com ORA-02068: following severe error from WEBDB.COMORA-03113: end-of-file on communication channelCurrent SQL statement for this session: SELECT ACCESS_LOG_SEQUENCE.NEXTVAL@WEBDB.COM from dual ----- PL/SQL Call Stack -- --- Object line object handle number name0x39b5c3720 5 ECOMMERCE. P_USER_AT ----- Call Stack Trace ----- calling call entry argument values in hex location type point (? Means dubious value) -------------------- ------------------------ ---------------------------- ksedst () + 31 call ksedst1 () 000000000? 000000001? 7FBFFF4370? 7FBFFF43D0? 7FBFFF4310? 000000000? Ksedmp () + 610 call ksedst () 000000000? 000000001? 7FBFFF4370? 7FBFFF43D0? 7FBFFF4310? 000000000? Ksfdmp () + 21 calling ksedmp () 000000003? 000000001? 7FBFFF4370? 7FBFFF43D0? 7FBFFF4310? 000000000 ?... 0059DF200? 683F6E400000001? Main () + 116 call opimai_real () 000000002? 7FBFFFF4E0? 000000004? 7FBFFFF478? 0059DF200? 683F6E400000001? _ Libc_start_main () call main () 000000002? 7FBFFFF4E0? + 219 000000004? 7FBFFFF478? 0059DF200? 683F6E400000001? _ Start () + 42 call _ libc_start_main () 0007139F8? 000000002? 7FBFFFF628? 0052B4BD0? 000000000? 000000002? Www.2cto.com --------------------- Binary Stack Dump --------------------- detailed TRACE analysis shows that the value of the remote sequence is being read through the database chain when a problem occurs. At this time the ORA-3113 communication error, most of the remote database status exception. Check the remote database alarm log, it is found that at the time of the problem, the database status is abnormal and eventually caused the instance restart: Wed Feb 29 19:39:29 2012 WARNING: inbound connection timed out (ORA-3136) wed Feb 29 19:39:30 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed out (ORA-3 136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:40:01 2012 WARNING: inbound connection timed out (ORA-3136 )... wed Feb 29 19:43:28 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:43:28 2012 www.2cto.com WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:43:28 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:43:28 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed out (ORA-3136) wed Feb 29 19:43:29 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:43:29 2012 WARNING: inbound connection timed o Ut (ORA-3136) Wed Feb 29 19:43:30 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:45:26 2012 PMON failed to acquire latch, see PMON dumpWed Feb 29 19:46:32 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:46:33 2012 WARNING: inbound connection timed out (ORA-3136) Wed Feb 29 19:46:34 2012 PMON failed to acquire latch, see PMON dumpWed Feb 29 19:46:40 2012 WARNING: inbo Und connection timed out (ORA-3136) Wed Feb 29 19:46:43 2012 WARNING: inbound connection timed out (ORA-3136) wed Feb 29 19:46:44 2012 Errors in file/opt/app/oracle/admin/orcl/bdump/orcl1_asmb_14614.trc: ORA-15064: communication failure with ASM instanceORA-03113: end-of-file on communication channelWed Feb 29 19:46:44 2012 www.2cto.com ASMB: terminating instance due to error 15064Wed Feb 29 19:46:44 2012 System state dump is made for local instanceSystem State dumped to trace file/opt/app/oracle/admin/orcl/bdump/orcl1_diag_14555.trcWed Feb 29 19:46:47 2012 Shutting down instance (abort) license high water mark = 1623Wed Feb 29 19:46:49 2012 Instance terminated by ASMB, pid = 14614Wed Feb 29 19:46:52 2012 Instance terminated by USER, pid = 3684 apparently the remote database status exception is the direct cause of this ORA-600 error.