How to Analyze problems related to Internal Errors (ORA-600) and Core dumps (ORA-7445) using My Oracle Support (document ID 2604 59.1)

Oracle database-enterprise edition-version 8.1.7.4 and later information in this document applies to any platform. **checked for relevance 06-apr-2010 **checked for relevance 17-apr-2013 * * Checked for relevance on 16-nov-2011 * * *

Purpose

1.1 Abstract ============ This document provides guidelines for customers to doing an initial analysis of problems related to Internal errors (ORA-600) and core dumps (ORA-7445) by using My Oracle support keyword searches. After finding a set of documents, either bug database entries or notes, these must is correlated to the specific Circumsta NCEs to further narrow down the search results. Hints to does this is given.
1.2 Introduction ================= It is often the case that certain problems has been already discovered and is documen Ted in My Oracle support notes and in published bug information. With the proper techniques it's often possible to narrow down to a particular bugs that matches your problem and find Docu mented workarounds or patch information. This document was mainly aimed at rediscovering bugs in Oracle code, cause internal errors or core dumps but it could be  Equally applicable to all kinds of problems. The following paragraphs aim at extracting the relevant keywords from the trace file to find the documents that is Releva NT to the specific error condition. It also tries to help in further narrowing down the list of relevant documents by correlating them to the specific circums Tances of the error.

Troubleshooting Steps

2.1 ora-00600:internal error code, arguments: [argument1] [Argumentx] .... =========================================== =============================== The internal argument ORA-600 is raised within the Oracle kernel when an exceptional Condi tion occurs. Inside the kernel code at various stages of processing, so called assertions is executed. These is certain conditions that must is true to being able to proceed. The assertions is internal health checks and guard over the integrity of memory and data of the instance and the database . When such an assertion fails, an ORA-600 error are raised with either a numeric or alphanumeric first argument and possibly More arguments depending on the particular error. Note that the ORA-600 errors is necessary fatal errors causing the session to terminate; Some is quite benign. Others however can be severe so they must always be carefully investigated.
2.1.1 The first Argument of the Internal Error ORA-600 ====================================================== the single m OST important piece of information is the first argument of the internal error, either numeric or alphanumeric; It uniquely identifies the specific module where it was raised and what assertion was failing. Always include this first argument in your keyword search when trying to rediscover known problems.
2.2 ora-07445:exception Encountered:core dump ============================================== A Core dump is an exception Al condition similar to the internal error ORA-600, however, the big difference are that the kernel do not anticipate the Error. Whereas in the internal error the exceptional condition were discovered by an assertion which is a predefined C Heck, the core dump happens because the operating system at some point aborts the process because it's doing a forbidden Action such as trying to access a area of memory that does not belong to the process. This is what core dumps is often referred to as access violations. The term ' core dump ' stems from a period when memory is stored with the use of magnetic cores, in computer terminology ' C Ore ' equates to ' memory '. A core dump means that memory of the process is dumped in a file ' core ' on the file system.
2.2.1 Identify the failing module ================================= It's important to know in which internal module the C Ore dump occurs, this is often printed together with the error and not always. If not refer to the sections 3.2.5 call Stack Trace. If known, include the failing module in your keyword search.
3.1 Trace files =============== for both internal errors and core dumps a Trace file are written in either User_dump_dest F or user processes and background_dump_dest for background processes. In 11g all trace files is written to the location defined by the parameter diagnostic_dest. The trace files for both types is treated in the same manner although the particular information in the trace files, whet Her a certain section is present, may depend on the particular error. In case of a core dump it's possible for the kernel to still dump relevant information by calling the dump routines such As Ksedmp () because the error is trapped by a signal handler. On the other hand, for most internal errors, the process was crashed (aborted) when the dump was ready.
3.2 Relevant sections in Trace files ==================================== and not intended to provide a detailed explana tion or in depth understanding, the following sections in the trace file can usually is identified, this'll help you Glean the relevant keywords for the search and understand the specific conditions under which the error occured, you shoul D then was able to narrow down your search results.
3.2.1 Header ============ The header section includes such information as the name of the trace file, the specific Oracle Version, Oracle_home, system name, node name, OS version, instance name and process information. While not directly relevant for your keyword search this information are vital later to correlate the documents to your pro Blem.
3.2.2 Error section =================== the error as it's written in the Alert.log is usually repeated, possibly along W ITH some extra information dependent on the error. When an internal error occurs a developer may has decided to write some crucial state information apart from the ORA-600 Arguments directly to the trace file. When the "the" case, those was usually gems for your search.
3.2.3 Current SQL statement for this session ============================================ "may or may not know" what is Being conducted at the time of the error. While common keywords like ' insert ', ' Update ' or ' delete ' is not being beneficial to your search (they is simply too common This is the course very useful to correlate documented rediscovery information to your problem. If For example a certain bug have in it rediscovery information that it happens on insert only and you are performing a de Lete statement, then it's safe to say the This bug was not your problem.
See also 3.2.6 Cursor Dump in case the current SQL statement are not present in the trace file.
3.2.4 PL/SQL call Stack ======================= This section is present if the session was performing pl/sql-it shows WHA T user or internal PL/SQL packages where called. Call Stacks is read bottom up; If there is Oracle packages in there include them in your keyword search.
3.2.5 Call stack trace ====================== the modules listed on the call stack trace provide excellent keywords that c An is used for rediscovery; They usually uniquely identify specific bugs. However, care must is taken to discard the top and bottom modules. Modules such as the following must be Ignored:sigacthandler (), Ssexhd (), Ksedmp (), Ksesicx () (where X was a number that D Esignates the number of extra arguments to an ora-600 besides the first). All kse* and kge* modules in general can is ignored, they stand for Kernel Service Error and Kernel Generic Error respecti vely; These is modules that is invoked after the error have occurred and perform such tasks as dumping the trace file and as Su CH is common to the most internal errors and core dumps so would not be in the narrowing down your search. In 11g onwards there is also dbk* modules which is part of the diagnostic dumping code. the same goes for the Botto M modules that is always present because they is used to initiateProcess startup; Everything below (and including) Opiexe () can safely be ignored. When you include three or four relevant modules from the top of the "Call stack" trace (together with some other relevant ke Ywords), this will often result in the relevant documents popping up to a My Oracle support search. If None is returned you could have to reduce the number of modules searched upon and removing the modules further down the Stack from the search. If you still has no results returned you could have the hit an as yet undiscovered problem.

3.2.6 Cursor Dump ================= Even when the current SQL statement was not listed on the top section of the the trace file Together with the error, the cursor dump can still reveal the SQL statement being performed at time of error. Simply search for ' current cursor ' identify the cursor number and scroll down until you find the cursor with that number. A bonus you also is able to identify the bind variables (if any) used for the statement on this section. See note:154170.1 for further information.
3.2.7 Process state-session State Object ========================================== a process state is a list of the pro Cess State objects. It is beyond the scope of this document to explain these in detail; For now it's enough to understand this state objects be used to organize memory objects that contain the relevant state Information of a session in an hierarchical manner. One of the state objects contains relevant session information that helps in narrowing down the specifics of the error con Dition. Simply Search for "program" in your trace file and you'll find a section similar to the following:
So:7000000223acfe0, Type:4, owner:700000022357868, flag:init/-/-/0x00 (session) TRANS:7000000230D47A0, creator:70000 0022357868, Flag: (18100041) usr/-bsy/-/-/-/-/-did:0001-0012-00000083, short-term did:0000-0000-00000000 txn branch:0 Oct:2, prv:0, Sql:70000002831a5a0, psql:70000002831a5a0, User:48/isis o/s info:user:someuser, Term:sometty01, OSP id:628:1948, machine:box\sometty01 program:someprogram.exe application Name:someprogram.exe, hash value=0 last wait fo R ' db file sequential read ' blocking sess=0x0 seq=1054 wait_time=16729 file#=1, BLOCK#=2EC3, Blocks=1 temporary object cou nter:0
When a specific program was being used, you could want to include this in your keyword search, whether it was an Oracle client program or not; We sometimes provide information on third party products in relation to Oracle on a ' as is ' basis. Try to identify what type of program is being used, these include JDBC (thin or OCI), pro*c, OCI etc. Some problems is for example specific to JDBC and this would help greatly in identifying the problem. From 11g onwards you can find this information by searching for the string ' client Details '.
4.1 Use "Advanced Search" ========================= to find relevant bugs database entries (more likely to contain call STA CK trace Modules) always perform the advanced search and make sure to check the ' Bugs Database ' check box from the ' sources ' section on the right of the page, in addition to the ' Knowledge Base ' which consists of the Notes written by Oracle Pers Onnel. Uncheck the ' Communities ' checkbox at first; When nothing was found in the Knowledge base and bugs database, some relevant info maybe found in the communities (this is n OT to downplay the communities, they just has a different use). The tips on the Advanced Search page provide further guidelines on how to search efficiently.
4.2 General Comments in Keyword searches ======================================== The important thing is trial and error. When a search returns an overwhelming mound of documents, try to narrow it down by including another keyword that's uniqu E to your problem. On the other hand, when no results is returned, omit a few; If you had included too many modules from a call stack trace, delete some from your search but retain the topmost module ( s); The specific module may has been called from a different one to your's when the bug is discovered or the call stack trac E was wasn't clearly documented in the bug.
4.3 No relevant Bug or Document is Found ======================================== your May's course be the first customer To has hit an as yet undiscovered problem. In this case, file a service request using My Oracle support and try to describe in detail "What's the problem is." You may also include a summary of the "The Analysis", which you have already performed based on these guidelines.
4.4 Provide Feedback ==================== if you find documents that is unclear or you think contain errors, or if you th Ink can add some relevant information based on your experience using our products, please use the feedback button and State the document number and your comment (use the "Technical Library feedback/questions ' radio button in step 2). We are very grateful for quality feedback as it improves our knowledge base.
5.1 Correlate Bugs and Documents to Your problem ================================================ now so you have gained More understanding of the problem by browsing through the trace file ' s relevant sections and has identified sufficiently Suitable keywords both in terms of quantity and quality (uniqueness) We have executed your search and is presented With some relevant documents. The Notes is usually clear enough; They go through a well defined process of QA and would provide you have detailed circumstances to match your problem.
Bugs can is more cumbersome to read; Try scrolling down to the bottom immediately (click "Go to End") to find the ' rediscovery ' or ' Release Note ' section (or S Earch for it) to match the bug description with your specifics. A closed published bug should has such a section unless it is a duplicate of another bugs. In this case, the base bug must be checked. For bugs resolved in 9.2 onwards there are often a summary note, with the reference bug number.8, which are easier to read.
You is hitting a certain bug if you can match all circumstances listed in the rediscovery information to your problem and The designated fix (either a patch or workaround) solves the problem.
6.0 Summary =========== Good search keywords include error messages, first arguments of internal errors and relevant Inter NAL module names. Program names and type could help narrow down further. Correlate with the documented rediscovery information. If unsure, file a service request with your findings, this helps the analyst.

References

note:153788.1-ora-600/ora-7445/ora-700 Error look-up Tool note:154170.1-how to Find the offending SQL from a Trace Fil E note:1812.1-tech:getting a Stack Trace from a CORE file on Unix