Why cannot I enable ADG for Oracle 11g slave database?

Why cannot I enable ADG for Oracle 11g slave database?

Today, we have encountered a strange problem, but there is an essential cause and effect behind the strange phenomenon.
During an Environment check in the afternoon, it was found that the standby database was in the mount stage. This was a 11gR2 database, and it was a waste of time without ADG. I felt too much to do this.
So I tried to start the open stage and found that the status was always read only. In the ADG, it should be READ ONLY WITH APPLY.
Use the dg broker to set READ-ONLY. The database logs of the standby database are as follows:
Standby Database: stestdb3, Enabled Physical Standby (0x02010000)
08/14/2014 16:03:28
Version check on database stestdb3 detected stale metadata,
Requesting update from primary database
Creating process RSM0
12/29/2015 16:28:11
Command edit database stestdb3 set state = READ-ONLY completed
Read-Only state no longer supported
12/29/2015 16:29:10
It seems that there is no clue. Use the dg broker to check the configuration. The following error is reported.
DGMGRL> show configuration;
Configuration-testdb
Protection Mode: MaxPerformance
Databases:
Testdbbak93-Primary database
Stestdb3-Physical standby database
Error: ORA-16766: Redo Apply is stopped
Fast-Start Failover: DISABLED
Configuration Status:
ERROR
View the log of the dg broker as follows:
Data Guard Broker initializing...
Data Guard Broker initialization complete
Tue Dec 29 16:47:15 2015
SMON: enabling cache recovery
No Resource Manager plan active
Physical standby database opened for read only access.
Completed: alter database open
Tue Dec 29 16:47:16 2015
Idle dispatcher 'd000' terminated, pid = (18, 1)
Tue Dec 29 16:51:40 2015
Primary database is in maximum performance mode
RFS [3]: Assigned to RFS process 3596
RFS [3]: Selected log 7 for thread 1 sequence 72606 dbid-1549369665 branch 746558785
Tue Dec 29 16:51:41 2015
RFS [4]: Assigned to RFS process 3590
RFS [4]: Selected log 8 for thread 1 sequence 72605 dbid-1549369665 branch 746558785
Tue Dec 29 16:51:42 2015
Archived Log entry 69432 added for thread 1 sequence 72605 ID 0xa829ec3b dest 2:
From the above situation, we can see that the MRP does not start to work, and only RFS receives the archive.
Then, use the dg broker to set the standby database to the ONLINE status, and check the check of the dg broker again. Then, the check shows that there is no problem.
DGMGRL> show configuration;
Configuration-testdb
Protection Mode: MaxPerformance
Databases:
Testdbbak93-Primary database
Stestdb3-Physical standby database
Fast-Start Failover: DISABLED
Configuration Status:
SUCCESS
Generally, this is not a 11g library.
Then try again, manually start to the open stage, and then you can see whether the slave database is read only. After the restart, the problem still exists.
For this problem, the best way is to view logs. This slave database was restarted a year ago. Fortunately, the database logs still exist. There were no other errors from the startup status at that time.
However, I have noticed the compatible parameter, because it is conspicuous in the 11g library. So this parameter aroused my curiosity.
The result was followed by a question in the MOS query. Several related articles appeared to have encountered another legacy problem and a related BUG description.
ACTIVE transaction uard (ADG) not possible with compatible <11.1.0.0.0 (Doc ID 1363396.1)
BUG: 13032521-ADG PHYSICAL STANDBY GOES TO MOUNT STATE INSTEAD OF READ ONLY WITH APPLY
After the problem is basically located, check that this parameter is 10.2.0.5.0 in the master and slave databases.
SQL> show parameter compa
NAME TYPE VALUE
-----------------------------------------------------------------------------
Compatible string 10.2.0.5.0
According to the bug description, WA sets the compatible of the slave database to 11.1.0.7 or above. This parameter needs to be modified to restart the instance, so it is still affected. Currently, the master database cannot be restarted.
SQL> alter system set compatible = '11. 2.0.3.0 ';
Alter system set compatible = '11. 2.0.3.0'
*
ERROR at line 1:
ORA-02095: specified initialization parameter cannot be modified
Now let's set up the slave database first.
SQL> alter system set compatible = '11. 2.0.3.0 'scope = spfile;
System altered.
When you restart, you can see that the database logs of the slave database have the following output.
Tue Dec 29 17:25:26 2015
Spfile/U01/app/Oracle/product/11.2.3/db_1/dbs/spfiletestdb. ora is in old pre-11 format and compatible> = 11.0.0; converting to new H. a.R. d. compliant format.
Completed: alter database mount
But set it to ONLINE again, and check whether the database status is still MOUNT.
SQL> select open_mode from v $ database;
OPEN_MODE
--------------------
READ ONLY
It seems that the slave database cannot be modified, and the master database must be modified as well.
However, you can see the following section in the database log and find that the MRP startup fails.
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Attempt to start background Managed Standby Recovery process (testdb)
Tue Dec 29 17:57:03 2015
MRP0 started with pid = 29, OS id = 17740
MRP0: Background Managed Standby Recovery process started (testdb)
Started logmerger process
Tue Dec 29 17:57:08 2015
Managed Standby Recovery starting Real Time Apply
Parallel Media Recovery started with 16 slaves
Waiting for all non-current ORLs to be archived...
All non-current ORLs have been archived.
Media Recovery Log/U01/app/oracle/fra/StestDB3/archivelog/2015_12_29/o1_mf_1_72606_c84n0xml _. arc
Completed: ALTER DATABASE RECOVER MANAGED STANDBY DATABASE THROUGH ALL SWITCHOVER DISCONNECT USING CURRENT LOGFILE
Errors with log/U01/app/oracle/fra/StestDB3/archivelog/2015_12_29/o1_mf_1_72606_c84n0xml _. arc
MRP0: Background Media Recovery terminated with error 38800
Errors in file/U01/app/oracle/diag/rdbms/stestdb3/testdb/trace/testdb_pr00_17745.trc:
ORA-38800: Cannot start Redo Apply on the open physical standby database
Managed Standby Recovery not using Real Time Apply
Recovery interrupted!
MRP0: Background Media Recovery process shutdown (testdb)
It seems that the impact of this parameter change is indeed not small. The slave database first restores the normal state, and then completes the process after the master database is restarted, so the original settings of the parameter are restored. Set compatible to 10.2.0.5.0
? However, an error is reported during the restart.
SQL> alter database mount;
Alter database mount
*
ERROR at line 1:
ORA-00201: control file version 11.2.0.3.0 incompatible with ORACLE version
10.2.0.5.0
ORA-00202: control file: '/U01/app/oracle/oradata/testdb/control01.ctl'
There seems to be room for this problem. When the master database generates the slave Database Control File and transmits the file, the mount will be okay.
Master database:
SQL> alter database create standby controlfile as '/tmp/std1.ctl ';
Database altered.
? Slave database:
SQL> alter database mount standby database;
Database altered.
However, when you check the database logs of the slave database, the problem may seem to be getting worse. The file header has been modified and is not synchronized.
Alter database recover managed standby database disconnect from session
Attempt to start background Managed Standby Recovery process (testdb)
Tue Dec 29 18:28:13 2015
MRP0 started with pid = 30, OS id = 24283
MRP0: Background Managed Standby Recovery process started (testdb)
Started logmerger process
Tue Dec 29 18:28:18 2015
Managed Standby Recovery not using Real Time Apply
Read of datafile '/U01/app/oracle/oradata/testdb/system01.dbf' (fno 1) header failed with ORA-01130
Rereading datafile 1 header failed with ORA-01130
MRP0: Background Media Recovery terminated with error 1110
Errors in file/U01/app/oracle/diag/rdbms/stestdb3/testdb/trace/testdb_pr00_24288.trc:
ORA-01110: data file 1: '/U01/app/oracle/oradata/testdb/system01.dbf'
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '/U01/app/oracle/oradata/testdb/system01.dbf'
ORA-01130: database file version 11.2.0.3.0 incompatible with ORACLE version 10.2.0.5.0
Slave exiting with ORA-1110 exception
Errors in file/U01/app/oracle/diag/rdbms/stestdb3/testdb/trace/testdb_pr00_24288.trc:
ORA-01110: data file 1: '/U01/app/oracle/oradata/testdb/system01.dbf'
ORA-01122: database file 1 failed verification check
ORA-01110: data file 1: '/U01/app/oracle/oradata/testdb/system01.dbf'
ORA-01130: database file version 11.2.0.3.0 incompatible with ORACLE version 10.2.0.5.0
Recovery Slave PR00 previusly exited with exception 1110
MRP0: Background Media Recovery process shutdown (testdb)
Completed: alter database recover managed standby database disconnect from session

The corresponding trace file is as follows:
* ** 18:28:18. 495 4320 krsh. c
Managed Standby Recovery not using Real Time Apply
Read of datafile '/U01/app/oracle/oradata/testdb/system01.dbf' (fno 1) header failed with ORA-01130
Rereading datafile 1 header failed with ORA-01130
V10 style file header:
Compatibility Vsn = 186647296 = 0xb200300
Db ID = 2745597631 = 0xa3a67ebf, Db Name = 'testdb'
Activation ID = 0 = 0x0
Control Seq = 1 = 0x1, File size = 147200 = 0x23f00
File Number = 1, blksiz= 8192, File Type = 3 DATA
Tablespace #0-SYSTEM rel_fn: 1
In this case, restoring the control file of the slave database for 11 GB should solve the problem by restarting the master database. However, restarting the master database requires coordination time to find the maintenance window, so it is not a one-stop task, disaster Tolerance is a top priority during this period. Once a problem occurs in the master database, the disaster recovery is not small. Therefore, the last helpless move is to re-build the slave database.
Of course, you can still use the 11g active mode to build a slave database.
Rman target sys @ xxxxx auxiliary sys @ xxxx nocatalog
RMAN> duplicate target database for standby from active database nofilenamecheck;
? Then there is no more, that is, the slave database is successfully built. Looking at a busy job, my heart is like turning over five flavors.