ORA-00600 [qertbfetchbyrowid]

Symptom:
1. crs_stat-T: The ONS service on both nodes is unavailable, and neither crs_start ora. longshi_rac01.ons nor crs_start-all can be started.
Ora... c01.ons application online offline long... ac01
Ora... c02.ons application online offline long... ac02

2. the hard disk of the server is suddenly used up. df-h shows that the usage rate is 100%.

3. sqlplus cannot log on normally

Analysis:
1. du-sh * | sort-Rn: searches for large files and finds that there are many trace files about MB in the $ oracle_base/admin/$ oracle_sid/bdump/directory, these files fill the hard disk.
2. After you use RM to delete these trace files, ONS can be started and sqlplus can be restored to normal.
3. The trace files under bdump are still being generated, and 34 GB files are generated in four hours.
4. bdump is the place where the tracking file is generated when an error occurs in the Oracle background process. It is determined that a problem occurs to the database.
5. view the. TRC file and find that all SQL statements cause ORA-00600 errors
6. Find the table with the error and find that the ORA-00600 [qertbfetchbyrowid]
7. Rebuild the index alter index idx_smsid rebuild on the table. The problem persists.
8. Recreate the table and index to solve the problem.

Below is a reference piece of article, the original http://www.anysql.net/oracle/ora_orphan_rowid.html
/**********
Oracle will certainly automatically maintain indexes when updating tables. However, due to a program bug (4258825,400 0840), after the insert/delete/update table records under some conditions, oracle does not maintain indexes. It seems like this. The table has five records (ID = 1, 2, 4, 5). At this time, the index is good, now, four records with ID = 5 are deleted, and five records are retained after Oracle does not maintain this index. When we issue a query such as where id = 5, the problem arises, because the records found based on rowid are incorrect. This is called orphan rowid.

In fact, such information will be recorded in alert_sid.log and rowid will be reported. The verification method is as follows:

Select/* + full (t) */count (*) from table t where ID is not null
Select/* + index (T, index name in the ID column) */count (*) from table t where ID is not null

If the results of these two queries are different (Please Query when no application changes to the table), it means there is a problem. You can re-create the index to solve this problem. generally, the number of records on the index is more than that of the table, which may not be observed in turn.

Today, a friend has encountered this kind of experience. Therefore, although Oracle says both bugs have been fixed in version 10.2.0.1, many bugs are fixed if they are not fixed, still.
**********/