"Panic" caused by redo logs when RMAN_RAC is restored to a single machine"

"Panic" caused by redo logs when RMAN_RAC is restored to a single machine"

I. Summary

In fact, this problem was caused by negligence. Previously, RAC was used to recover data from a single machine. The production environment was not involved. At that time, the testing environment was RedHat5.4 + Oracle11g + RAC, the storage mode is ASM, the production environment is hpux + Oracle10g + RAC, and the storage mode is bare devices. Currently, it depends on the storage mode of data files.

2. In Hpux, the recover is complete. The following error is reported when 'alter database open resetlogs' is executed.

SQL> alter database open resetlogs;

Alter database open resetlogs

*

ERROR at line 1:

ORA-19502: write errors on file "/dev/vgdb/rredo2_2a_256m", blockno 105473

(Blocksize = 1024)

ORA-27072: File I/O error.

HPUX-ia64 Error: 2: No such file or directory

Additional information: 4

Additional information: 105473

Additional information: 433152
 

 

View alert logs

 

ORA-1589 signalled during: alter database open...

Weds Oct 29 08:15:21 2014

Alter database open resetlogs

Weds Oct 29 08:15:21 2014

Errors in file/oracle/admin/pmssd/udump/pmssd1_ora_11597.trc:

ORA-00313: open failed for members of log group 1 of thread 1

ORA-00312: online log 1 thread 1: '/dev/vgdb/rredo1_1b_256m'

ORA-27037: unable to obtain file status

HPUX-ia64 Error: 2: No such file or directory

Additional information: 3

ORA-00312: online log 1 thread 1: '/dev/vgdb/rredo1_1ax' _ 256m'

ORA-27037: unable to obtain file status

HPUX-ia64 Error: 2: No such file or directory

Additional information: 3

Weds Oct 29 08:15:21 2014
 

 

View view v $ logfile

 

SQL> select * from v $ logfile;

 

GROUP # status type member is _

-------------------------------------------------------

1 ONLINE/dev/vgdb/rredo1_1a_256m NO

2 ONLINE/dev/vgdb/rredo1_2a_256m NO

3 ONLINE/dev/vgdb/rredo2_1a_256m NO

4 ONLINE/dev/vgdb/rredo2_2a_256m NO

1 ONLINE/dev/vgdb/rredo1_1b_256m NO

2 ONLINE/dev/vgdb/rredo1_2b_256m NO

3 ONLINE/dev/vgdb/rredo2_1b_256m NO

4 ONLINE/dev/vgdb/rredo2_2b_256m NO

5 ONLINE/dev/vgdb/rredo1_3a_256m NO

5 ONLINE/dev/vgdb/rredo1_3b_256m NO

6 ONLINE/dev/vgdb/rredo2_3a_256m NO

6 ONLINE/dev/vgdb/rredo2_3b_256m NO
 

 

The dev directory under the root directory is the Set directory. We need to modify the redo log directory to complete the open database operation. The statement is as follows:

 

SQL> alter database rename file '/dev/vgdb/rredo1_1a_256m' to'/oracle/oradata/pmssd/redo01.log ';
 

 

I did not follow the above statements, but re-created the control file. This problem was caused by my own mistake. After modifying the redo log directory, you can open the database.

The following is my operation in linux ASM mode ('alter database open resetlogs '). As shown in the following logs, the database automatically recreates the redo log and specifies the corresponding directory.

 

Weds Oct 29 09:27:19 2014

Alter database open resetlogs

RESETLOGS after incomplete recovery until change 783268

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3900.trc:

ORA-00313: open failed for members of log group 1 of thread 1

ORA-00312: online log 1 thread 1: '+ DATA/racdb/onlinelog/group_1.257.859442553'

ORA-17503: ksfdopn: 2 Failed to open file + DATA/racdb/onlinelog/group_1.257.859442553

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15077: cocould not locate ASM instance serving a required diskgroup

ORA-29701: unable to connect to Cluster Synchronization Service

ORA-00312: online log 1 thread 1: '+ DATA/racdb/onlinelog/group_1.273.859438547'

ORA-17503: ksfdopn: 2 Failed to open file + DATA/racdb/onlinelog/group_1.273.859438547

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15077: cocould not locate ASM instance serving a required diskgroup

ORA-29701: unable to connect to Cluster Synchronization Service

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_3900.trc:

......................................................................

WARNING: ASM communication error: op 16 state 0x40 (15077)

ERROR: slave communication error with ASM

WARNING: Cannot delete Oracle managed file + DATA/racdb/onlinelog/group_4.275.859442003

WARNING: ASM communication error: op 16 state 0x40 (15077)

ERROR: slave communication error with ASM

WARNING: Cannot delete Oracle managed file + DATA/racdb/onlinelog/group_4.271.859442011

Weds Oct 29 09:28:15 2014

Clearing online redo logfile 4 complete

Resetting resetlogs activation ID 857045326 (0x3315794e)

Online log/oracle/oradata/racdb/RACDB/onlinelog/ow.mf_rjb50jmr0p _. log: Thread 1 Group 1 was previusly cleared

Online log/oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_1_b50jmsl4 _. log: Thread 1 Group 1 was previusly cleared

Online log/oracle/oradata/racdb/RACDB/onlinelog/o1_mf_2_b50jn5rw _. log: Thread 1 Group 2 was previusly cleared

Online log/oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_2_b50jn6jm _. log: Thread 1 Group 2 was previously cleared

Online log/oracle/oradata/racdb/RACDB/onlinelog/ow.mf_3_b50jnkqn _. log: Thread 2 Group 3 was previusly cleared

Online log/oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_3_b50jnltx _. log: Thread 2 Group 3 was previusly cleared

Online log/oracle/oradata/racdb/RACDB/onlinelog/o1_mf_4_b50jnz10 _. log: Thread 2 Group 4 was previously cleared

Online log/oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_4_b50jo0jb _. log: Thread 2 Group 4 was previusly cleared

Weds Oct 29 09:28:16 2014

Setting recovery target incarnation to 2

Weds Oct 29 09:28:17 2014

Assigning activation ID 859863386 (0x3340795a)

LGWR: STARTING ARCH PROCESSES

Weds Oct 29 09:28:17 2014

ARC0 started with pid = 20, OS id = 3924

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

Thread 1 opened at log sequence 1

Current log #1 seq #1 mem #0:/oracle/oradata/racdb/RACDB/onlinelog/o1_mf_1_b50jmr0p _. log

Current log #1 seq #1 mem #1:/oracle/backup/archive_racdb/RACDB/onlinelog/o1_mf_1_b50jmsl4 _. log

Successful open of redo thread 1

ARC0: STARTING ARCH PROCESSES

Weds Oct 29 09:28:17 2014

ARC1 started with pid = 21, OS id = 3926

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

Weds Oct 29 09:28:17 2014

SMON: enabling cache recovery

Weds Oct 29 09:28:17 2014

ARC2 started with pid = 22, OS id = 3928

Weds Oct 29 09:28:17 2014

ARC3 started with pid = 23, OS id = 3930

ARC1: Archival started

ARC2: Archival started

ARC1: Becoming the 'no fal' ARCH

ARC1: Becoming the 'no srl' ARCH

ARC2: Becoming the heartbeat ARCH

ARC3: Archival started

ARC0: STARTING ARCH PROCESSES COMPLETE

Redo thread 2 internally disabled at seq 1 (CKPT)

ARC0: Archiving disabled thread 2 sequence 1

Archived Log entry 27 added for thread 2 sequence 1 ID 0x0 dest 1:

[2, 3900] Successfully onlined Undo Tablespace 2.

Undo initialization finished serial: 0 start: 4182896 end: 4184866 diff: 1970 (19 seconds)

Dictionary check beginning

File #7 is offline, but is part of an online tablespace.

Data file 7: '+ DATA/racdb/datafile/test2.260.859451699'

File #9 is offline, but is part of an online tablespace.

Data file 9: '+ DATA/racdb/datafile/test4.258.859451727'

Weds Oct 29 09:28:24 2014

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3814.trc:

ORA-01157: cannot identify/lock data file 201-see DBWR trace file

ORA-01110: data file 201: '+ DATA/racdb/tempfile/temp.286.8594000005'

ORA-17503: ksfdopn: 2 Failed to open file + DATA/racdb/tempfile/temp.286.859437905

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15077: cocould not locate ASM instance serving a required diskgroup

ORA-29701: unable to connect to Cluster Synchronization Service

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3814.trc:

ORA-01186: file 201 failed verification tests

ORA-01157: cannot identify/lock data file 201-see DBWR trace file

ORA-01110: data file 201: '+ DATA/racdb/tempfile/temp.286.8594000005'

File 201 not verified due to error ORA-01157

Dictionary check complete

Verifying file header compatibility for 11g tablespace encryption ..

Verifying 11g file header compatibility for tablespace encryption completed

SMON: enabling tx recovery

Re-creating tempfile + DATA/racdb/tempfile/temp.286.85942.1605 as/oracle/oradata/racdb/RACDB/datafile/o1_mf_temp_b50jor8m _. tmp

Database Characterset is ZHS16GBK

Weds Oct 29 09:28:27 2014

No Resource Manager plan active

Replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

Weds Oct 29 09:28:31 2014

QMNC started with pid = 24, OS id = 3937

LOGSTDBY: Validating controlfile with logical metadata

LOGSTDBY: Validation complete

Weds Oct 29 09:28:42 2014

Db_recovery_file_dest_size of 5727 MB is 13.97% used. This is

User-specified limit on the amount of space that will be used by this

Database for recovery-related files, and does not reflect the amount

Space available in the underlying filesystem or ASM diskgroup.

Weds Oct 29 09:28:45 2014

Starting background process CJQ0

Weds Oct 29 09:28:45 2014

CJQ0 started with pid = 27, OS id = 3957

Weds Oct 29 09:28:46 2014

Completed: alter database open resetlogs
 

 

Before 'alter database open resetlogs' is executed, modify the redo log directory and generate the redo log in the specified directory.

 

Alter database rename file '+ DATA/racdb/onlinelog/group_1.273.859438547' to '/oracle/oradata/racdb/redow.a.log ';

Alter database rename file '+ DATA/racdb/onlinelog/group_1.257.8594000053' to '/oracle/oradata/racdb/redow. B .log ';

Alter database rename file '+ DATA/racdb/onlinelog/group_2.256.859438561' to '/oracle/oradata/racdb/redo2_a.log ';

Alter database rename file '+ DATA/racdb/onlinelog/group_2.291.859438567' to '/oracle/oradata/racdb/redo2_ B .log ';

Alter database rename file '+ DATA/racdb/onlinelog/group_3.277.859441989' to '/oracle/oradata/racdb/redo3_a.log ';

Alter database rename file '+ DATA/racdb/onlinelog/group_3.276.859441997' to '/oracle/oradata/racdb/redo3_ B .log ';

Alter database rename file '+ DATA/racdb/onlinelog/group_4.275.859442003' to '/oracle/oradata/racdb/redo4_a.log ';
 

 

View alert logs

 

Lter database open resetlogs

RESETLOGS after incomplete recovery until change 783268

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_1_3.trc:

ORA-00313: open failed for members of log group 1 of thread 1

ORA-00312: online log 1 thread 1: '/oracle/oradata/racdb/redow. B .log'

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

ORA-00312: online log 1 thread 1: '/oracle/oradata/racdb/redow.a.log'

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_1_3.trc:

ORA-00313: open failed for members of log group 2 of thread 1

ORA-00312: online log 2 thread 1: '/oracle/oradata/racdb/redo2_ B .log'

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

...................................

Additional information: 3

ORA-00312: online log 2 thread 1: '/oracle/oradata/racdb/redo2_a.log'

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

Clearing online redo logfile 2/oracle/oradata/racdb/redo2_a.log

Clearing online log 2 of thread 1 sequence number 30

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_ora_1_3.trc:

ORA-00313: open failed for members of log group 2 of thread 1

ORA-00312: online log 2 thread 1: '/oracle/oradata/racdb/redo2_ B .log'

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

......................................

Clearing online redo logfile 4 complete

Resetting resetlogs activation ID 857045326 (0x3315794e)

Online log/oracle/oradata/racdb/redow.a.log: Thread 1 Group 1 was previusly cleared

Online log/oracle/oradata/racdb/redow. B .log: Thread 1 Group 1 was previusly cleared

Online log/oracle/oradata/racdb/redo2_a.log: Thread 1 Group 2 was previusly cleared

Online log/oracle/oradata/racdb/redo2_ B .log: Thread 1 Group 2 was previusly cleared

Online log/oracle/oradata/racdb/redo3_a.log: Thread 2 Group 3 was previusly cleared

Online log/oracle/oradata/racdb/redo3_ B .log: Thread 2 Group 3 was previusly cleared

Online log/oracle/oradata/racdb/redo4_a.log: Thread 2 Group 4 was previusly cleared

Online log/oracle/oradata/racdb/redo4_ B .log: Thread 2 Group 4 was previusly cleared

Weds Oct 29 09:18:08 2014

Setting recovery target incarnation to 2

Weds Oct 29 09:18:09 2014

Assigning activation ID 859838345 (0x33401789)

LGWR: STARTING ARCH PROCESSES

Weds Oct 29 09:18:09 2014

ARC0 started with pid = 20, OS id = 3730

ARC0: Archival started

LGWR: STARTING ARCH PROCESSES COMPLETE

ARC0: STARTING ARCH PROCESSES

Thread 1 opened at log sequence 1

Current log #1 seq #1 mem #0:/oracle/oradata/racdb/redow.a.log

Current log #1 seq #1 mem #1:/oracle/oradata/racdb/redow. B .log

Successful open of redo thread 1

Weds Oct 29 09:18:10 2014

ARC1 started with pid = 21, OS id = 3732

MTTR advisory is disabled because FAST_START_MTTR_TARGET is not set

Weds Oct 29 09:18:10 2014

SMON: enabling cache recovery

Weds Oct 29 09:18:10 2014

ARC2 started with pid = 22, OS id = 3734

Weds Oct 29 09:18:10 2014

ARC3 started with pid = 23, OS id = 3736

ARC1: Archival started

ARC2: Archival started

ARC3: Archival started

ARC0: STARTING ARCH PROCESSES COMPLETE

ARC0: Becoming the 'no fal' ARCH

ARC0: Becoming the 'no srl' ARCH

ARC2: Becoming the heartbeat ARCH

Redo thread 2 internally disabled at seq 1 (CKPT)

ARC0: Archiving disabled thread 2 sequence 1

Archived Log entry 27 added for thread 2 sequence 1 ID 0x0 dest 1:

Weds Oct 29 09:18:22 2014

[2, 3683] Successfully onlined Undo Tablespace 2.

Undo initialization finished serial: 0 start: 3581386 end: 3583566 diff: 2180 (21 seconds)

Dictionary check beginning

File #7 is offline, but is part of an online tablespace.

Data file 7: '+ DATA/racdb/datafile/test2.260.859451699'

File #9 is offline, but is part of an online tablespace.

Data file 9: '+ DATA/racdb/datafile/test4.258.859451727'

Weds Oct 29 09:18:23 2014

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3455.trc:

ORA-01157: cannot identify/lock data file 201-see DBWR trace file

ORA-01110: data file 201: '+ DATA/racdb/tempfile/temp.286.8594000005'

ORA-17503: ksfdopn: 2 Failed to open file + DATA/racdb/tempfile/temp.286.859437905

ORA-15001: diskgroup "DATA" does not exist or is not mounted

ORA-15077: cocould not locate ASM instance serving a required diskgroup

ORA-29701: unable to connect to Cluster Synchronization Service

Errors in file/oracle/app/oracle/diag/rdbms/racdb/racdb1/trace/racdb1_dbw0_3455.trc:

ORA-01186: file 201 failed verification tests

ORA-01157: cannot identify/lock data file 201-see DBWR trace file

ORA-01110: data file 201: '+ DATA/racdb/tempfile/temp.286.8594000005'

File 201 not verified due to error ORA-01157

Dictionary check complete

Verifying file header compatibility for 11g tablespace encryption ..

Verifying 11g file header compatibility for tablespace encryption completed

Weds Oct 29 09:18:23 2014

SMON: enabling tx recovery

Re-creating tempfile + DATA/racdb/tempfile/temp.286.85942.1605 as/oracle/oradata/racdb/RACDB/datafile/o1_mf_temp_b50j2zdh _. tmp

Database Characterset is ZHS16GBK

No Resource Manager plan active

Replication_dependency_tracking turned off (no async multimaster replication found)

Starting background process QMNC

Weds Oct 29 09:18:32 2014

QMNC started with pid = 24, OS id = 3743

Weds Oct 29 09:18:33 2014

LOGSTDBY: Validating controlfile with logical metadata

LOGSTDBY: Validation complete

Weds Oct 29 09:18:45 2014

Starting background process CJQ0

Weds Oct 29 09:18:45 2014

CJQ0 started with pid = 27, OS id = 3763

Weds Oct 29 09:18:45 2014

Completed: alter database open resetlogs
 

Iii. Summary
It can be seen from the above that when the Oracle database is restored (that is, when 'alter database open resetlogs' is executed), if the storage mode of the Oracle database is ASM, when the database is opened, the database automatically creates redo logs in the corresponding directory. When the storage mode of the Oracle database is raw device or file mode, when the database is opened, the database will find the original raw device or directory, and create the original directory structure. If it cannot be created, an error is returned. To change the value, you must manually set the value. It can be seen that automatic management of ASM is very good. Of course, as a technician, we can be more rigorous (automatic stuff is sometimes unreliable), and modify the redo log directory before executing open. Before executing hpux, I have checked the operation commands and related documents repeatedly, and they will always be very confidential. Of course, this problem is not found in the restored ASM mode, in addition, the contingency and emergency response capabilities need to be improved.

-------------------------------------- Recommended reading --------------------------------------

RMAN: Configure an archive log deletion policy

Basic Oracle tutorial-copying a database through RMAN

Reference for RMAN backup policy formulation

RMAN backup learning notes

Oracle Database Backup encryption RMAN Encryption

-------------------------------------- Split line --------------------------------------