Oracle Data Recovery: formatting, ASM, and dictionary Damage Case 3

Link: http://www.eygle.com/archives/2010/06/asm_format_dictionary.html

 

In the last week, I have helped multiple users recover data and saved databases in distress. I would like to share with you the following cases:

Case 1: Misoperation and formatting during storage maintenance
A hard disk in use, causing database crash
.
After formatting, you have created an ext3 file system, which makes the situation more complex.

Customer
The user's original system uses the ASM storage management, which is a storage device of about 2 TB consisting of two hard disks. In this way, we must reorganize the data through two hard disks during recovery, the default au size of ASM is 1 m,
Balancing between two disks is a performance improvement technology in Oracle. However, in the case of a fault, you will find this technology to be tough, usually using file systems.
System, a file is stored on a single system, and ASM is scattered, which leads to even the first system tablespace created, it must be stored alternately between two disks.

Good
So what we should learn is:Be careful when maintaining disks. When necessary, use tools to compare disk partitions. I am used to compare Disk Partitions with ue.
.

Case
Example 2: In the RAID 5 disk array, two hard disks are lost instantly. data inconsistency occurs after the disk is forcibly released, and the database cannot be started.

The initial error message is the loss of the redo log.
Bad. Further, ASM cannot be attached normally. The background rbal process sometimes shows as a deadlock.

In the final verification, we found that multiple data files were damaged, that is, disk loss.
Bad and load results in damage to multiple files. The database is non-archive and has no backup. As the data volume is at the TB level, recovery is troublesome.

In the process of processing this case, I have determined that
I have summarized some new DBA rules and wrote a PPT, "dba misjudgment ", in the future, you will have the opportunity to share it with acoug.

Most
Then we instruct the user to extract and restore the data through tools.

Case 3: frequent creation and deletion of tablespaces results in inconsistent data dictionaries and database failures
Regular Operation
.
I think this is a bug in Oracle. Although there is no mark on Metalink, it is an oracle issue. The final manifestation of the fault is:
Tablespace files are displayed normally, but the deletion prompt does not exist, and other operations are affected.
The following error occurs during the initial tablespace drop operation:

Thu
Jun 24 20:00:04 2010
Drop tablespace FMI
Thu Jun 24 20:00:04 2010
ORA-959
Signalled during: Drop tablespace FMI

Then the following ORA-00600 error occurs
Error:

Thu Jun 24 20:03:59 2010
Errors in file
/Oracle/admin/cgk/udump/cgk_ora_25919.trc:
ORA-00600: Internal error
Code, arguments: [4348], [u], [0], [229], [], [], [], []

4348
The error code is not recorded on Metalink. In this case, the corresponding 229 tablespace cannot be deleted.

In further repair, the database showed ora-
Errors 00600 and 25015 of 25015 are related to the subsequent tablespace files.

Fri Jun 25 09:18:40
2010
Errors in file/Oracle/admin/cwg/udump/cwgw.ora_20031.trc:
ORA-00600:
Internal error code, arguments: [25013], [0], [229], [feen2], [Feek],
[184], [179], []
Fri Jun 25 09:18:45 2010
Errors in file
/Oracle/admin/cwg/bdump/cwg1_pmon_4050.trc:
ORA-00600: Internal error
Code, arguments: [25015], [229], [179], [184], [], [], [], []
Fri
Jun 25 09:18:47 2010
Trace dumping is refreshing Ming
Id = [cdmp_20100625091847]
Fri Jun 25 09:18:47 2010
Errors in file
/Oracle/admin/cwg/bdump/cwg1_pmon_4050.trc:
ORA-00600: Internal error
Code, arguments: [kccocx_01], [], [], [], [], [], [], []
ORA-00600:
Internal error code, arguments: [25015], [229], [179], [184], [], [],
[], []

Under normal circumstances, this problem can be corrected through the offline/drop operations of the tablespace file, but the customer tried
A lot of recovery methods, and eventually the database cannot be started.
After the control file is re-created and the corresponding file is cleared, the database fails with errors 2662 and 2663:

Sat
Jun 26 22:28:49 2010
Errors in file
/Oracle/admin/cwg/udump/cwg1_ora_23293.trc:
ORA-00600: Internal error
Code, arguments: [2662], [0], [487169572], [0], [487170770], [4194313],
[], []
Sat Jun 26 22:28:51 2010
Errors in file
/Oracle/admin/cwg/udump/cwg1_ora_23293.trc:
ORA-00600: Internal error
Code, arguments: [2662], [0], [487169572], [0], [487170770], [4194313],
[], []

Sun Jun 27 21:21:52 2010
Errors in file
/Oracle/admin/cwg/udump/cwg1_ora_3887.trc:
ORA-00600: Internal error
Code, arguments: [1, 2663], [0], [487192946], [0], [487202512], [], [], []
Sun
Jun 27 21:21:57 2010
Errors in file
/Oracle/admin/cwg/udump/cwg1_ora_3887.trc:
ORA-00600: Internal error
Code, arguments: [1, 2663], [0], [487192946], [0], [487202512], [], [], []

Connect
You can forcibly open the database by using the _ allow_resetlogs_uption parameter and 10015 event:
Alter session
Set events '10015 trace name adjust_scn level ';

Of course, this situation may cause data loss.
Loss.

After the database is forcibly opened through these methods, you can manually modify the table space information and file $ error records to restore the normal operation of the database.

It seems that
A database fault occurs frequently. Therefore, ask the DBAs to back up the database in a timely and effective manner and exercise caution when performing any database operations.

 

Website related articles | related articlesORA-00600
Kcratr_nab_less_than_odr case 1

SMON:
Recover undo segment and transaction recovery

ORA-600
Kcbzpbuf_1 case 1

Use
Ora_rowscn identifies time points of misoperation data

Disconnected
Fault and recovery case of ASM diskgroup caused by electrical fault