Oracle ORA-00600 [15764] Solution

When I arrived at the company, I received a friend's message saying that one node of RAC had crashed. Because he re-built an index at last night and ran for more than two hours before it was over, then he manually canceled it. At, one of the nodes encountered a problem.

The DB environment is: AIX 6.1 + Oracle 10.2.0.4, 2 nodes. Now a node is under great pressure. If you try to start a failed node, it can be started normally. Once you perform the DML operation, the node will be suspended.

Alert log information:

ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Wed Sep 7 00:26:42 2011
Errors in file/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Wed Sep 7 00:26:44 2011
Trace dumping is refreshing Ming id = [cdmp_20110907002644]
Wed Sep 7 00:27:10 2011
Errors in file/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_2756730.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Wed Sep 7 00:28:11 2011
Errors in file/apps/oracle/admin/sfc3db/udump/sfc3db2_ora_2756730.trc:
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []

I asked my friend if there were any other errors, and they said they were the only ones.

Some Trace information is as follows:

/Apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: CILS aborted: 0, num est exec limit hit: 0
/Apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: name = update seq $ setincrement $ =: 2, minvalue =: 3, maxvalue =: 4, cycle #=: 5, order $ =: 6, cache =: 7, highwater =: 8, audit $ =: 9, flags =: 10 where obj # =: 1
/Apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: name = selectdecode (failover_method, NULL, 0, 'Basic ', 1, 'preconnect', 2, 'preparse ', 4, 0), decode (failover_type, NULL, 1, 'none', 1, 'session ', 2, 'select', 4, 1), failover_retries, failover_delay, flags from service $ where name =: 1
/Apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: selectSYS_CONTEXT ('userenv', 'server _ host'), SYS_CONTEXT ('userenv', 'db _ UNIQUE_NAME '), SYS_CONTEXT ('userenv', 'instance _ name'), SYS_CONTEXT ('userenv', 'service _ name'), INSTANCE_NUMBER, STARTUP_TIME, SYS_CONTEXT ('userenv ', 'db _ DOMAIN ') from v $ instance whereINSTANCE_NAME = SYS_CONTEXT ('userenv', 'instance _ name ')
/Apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: name = select value $ from props $ where name = 'Global _ DB_NAME'
/Apps/oracle/admin/sfc3db/udump/dumps: name = selectprivilege #, level from sysauth $ connect by grantee # = prior privilege # andprivilege #> 0 start with grantee # =: 1 and privilege #> 0
/Apps/oracle/admin/sfc3db/udump/sfc3db2_ora_5054706.trc: name = select privilege # from sysauth $ where (grantee # =: 1 or grantee # = 1) and privilege #> 0

I have prepared an article:

Description of parameters of ORA-600

Based on the description in this article, you can determine:

ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []

Is related to parallel queries.

My friend asked me if I had something to do with the interrupted index rebuild. I said no.

Some temporary segments will be left behind for manual re-indexing interrupted. During the rebuild index, a temporary segment will be allocated in the segments of the user's index space to save the index information. After the rebuild is complete, the old index is droped, the previously assigned temporary segments are defined as permanent segment.

If the rebuild operation is interrupted, SMON will be cleared once every three minutes (only if it is received post). If SMON is too busy, temporary segment may not be cleared for a long time. Temporary segment is not cleared for a long time, which may cause a typical problem: After rebuild index online fails, the subsequent rebuild index Command requires that the generated temporary segment has been cleanup, if cleanup is not completed, you need to wait.

The following two articles describe in detail:

Oracle alter index rebuild description

Oracle rebuild index reporting ORA-01652 Solution

I am still positioning the problem on the SQL statement of parallel query. However, the trace provided by a friend does not obtain relevant information.

I searched on MOS and found a related article:

ORA-600 [15764] Running Parallel Query on RAC [ID839536.1]

An ORA-600 [15764] is highly transient in nature.Most bugs filed for thisissue have been closed as not reproducible.The purposeof this note is to document a known workaround if you see this error withsimilar circumstances.

The followingselect statement failed in a 10.2.0.3, 3-node, RAC database when runningin parallel:

Ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [15764], [], [], [], [], [], [], []
Current SQL statement for this session:
SELECT/* + PARALLEL (ORG, 3, 3 )*/
ORG. X_EFX_SSN,
ORG. X_EFX_BIRTH_DT,
COUNT (*)
FROM
SIEBEL. S_ORG_EXT ORG
GROUP
X_EFX_SSN,
X_EFX_BIRTH_DT
Having count (*)> 1

The tracefile showed the following callstack:

----- Call Stack Trace -----
Signature <-signature <-kxfxgs <-kxfxcw <-qerpxFetch <-opifch2 <-kpoal8 <-opiodr <-ttcpip <-opitsk <-opiino <-opiodr <-opidrv <-sou2o <-opimai_real <-main <-start

 

The tracefile also showed the process statewas busy holding a child latch:

========================================================== ==============
PROCESS STATE
-------------
Process global information:
Process: 7000004748eaa00, call: 700000443d33870, xact: 0, curses: 700000474e203c8, usrses: 700000474e203c8
----------------------------------------
SO: 7000004748eaa00, type: 2, owner: 0, flag: INIT/-/0x00
(Process) Oracle pid = 157, callcur/top: 700000443d33870/700000443d33870, flag: (0 )-
Int error: 0, call error: 0, sess error: 0, txn error 0
(Post info) last post received ed: 0 0 249
Last post received-location: kxfprienq: QC
Last process to post me: 700000474914f40 186 0
Last post sent: 0 0 250
Last post sent-location: kxfprienq: slave
Last process posted by me: 700000474914f40 186 0
(Latch info) wait_event = 0 bits = 10
Holding (efd = 7) 700000472e4a838 Child process queue reference level = 4 child # = 99
Location from where latch is held: kxfprigdb: KSLBEGIN: addr qref <---
Context saved from call: 504403177372952360
State = busy, wlstate = free <----
Process Group: DEFAULT, pseudo proc: 700000474a384f0
O/S info: user: oracle, term: UNKNOWN, ospid: 2351144
OSD pid info: Unix process pid: 2351144, image: oraclePPSOLTP1 @ psoldbap003

The final solution:

Workaround:

Bounce all instances in the RAC cluster.

Restart all instances on RAC. A friend applied for the downtime at noon. After the restart, the RAC node is normal.