Oracle 11.2.0.1 ADG Environment MRP process encounters Ora

Environment: Linux + Oracle 11.2.0.1 ADG
Symptom: Discovery Repository has no application log

1. Database query Repository current status
There is currently no application log for the repository, and apply lag has shown that the repository has no application logs for 3 days and 21 hours.

Sql> set linesize 1200sql> SELECT open_mode, Database_role, Switchover_status, force_logging, DATAGUARD_BROKER, Guard_status from V$database; Open_mode database_role switchover_status for Dataguar guard_s----------------------------------------- ---------------------------------READ only physical STANDBY not allowed YES DISABLED nonesql> SE                                                            Lect * from V$dataguard_stats;name VALUE UNIT time_computed datum_time------------------------------------------- ----------------------------------------------------- ------------------------------ ----------------------------                                                     --------------------------------transport Lag +00 00:00:00                        Day (2) to second (0) interval 01/17/2017 16:07:12 01/17/2017 16:07:12apply Lag +03 21: 34:49 Day (2) to second (0) interval 01/17/2017 16:07:12 0  1/17/2017 16:07:12apply finish Time +00 03:10:34.000 Day (2)                                                               to second (3) interval 01/17/2017 16:07:12estimated startup time 15 Second 01/17/2017 16:07:12

2. Query Alert alarm log
from alert alarm log to locate the ADG in the event of a problem, there are 600 error messages, resulting in the MRP process terminated, detailed log as follows:

Fri Jan 18:32:25 2017Errors in File/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr03_22555.trc (  incident=67480): ora-00600:internal error code, arguments: [Kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []incident details in:/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/incident/incdir_67480/orcl_pr03_22555_ I67480.trcslave exiting with ORA-600 exceptionerrors in File/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_ Pr03_22555.trc:ora-00600:internal error code, arguments: [Kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []fri Jan 18:32:26 2017Errors in File/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_mrp0_22547.trc (  incident=67448): ora-00600:internal error code, arguments: [Kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []incident details in:/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/incident/incdir_67448/orcl_mrp0_22547_ I67448.trcfri Jan 18:32:26 2017Trace dumping is performing id=[cdmp_20170113183226]recovery Slave PR03 previously exited with exception 600Fri-Jan 18:32:27 2017mrp0:background Media Recovery t Erminated with error 448Errors in File/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_22549.trc:o Ra-00448:normal completion of background processmanaged Standby Recovery not using Real time Applyrecovery interrupted! Fri Jan 18:32:27 2017Sweep [inc][67480]: Completedsweep [inc][67480]: completedrecovered data files to a consistent STA Te at change 2010287982Errors in File/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/orcl_pr00_22549.trc:o Ra-00448:normal completion of background processerrors in file/home/oracle/opt/oracle/diag/rdbms/orcl/orcl/trace/  Orcl_mrp0_22547.trc:ora-00600:internal error code, arguments: [Kcbr_apply_change_11], [], [], [], [], [], [], [], [], [], [], []mrp0:background Media Recovery process shutdown (ORCL) Sweep [inc][67448]: Completedsweep [inc2][67480]: completeds Weep [inc2][67448]: completedtrace dumping is performing Id=[cdmp_20170113183227]fri Jan 18:33:04 2017Using standby_archive_dest parameter default value as Use_db_recovery_fi Le_dest

3. Try to manually start the standby MRP recovery process
It is found that the same ORA-600 [Kcbr_apply_change_11] error is still reported in the alarm log when the MRP recovery process is started manually.

4. Try the mount state to start the MRP recovery process
Found in the Mount state, can normally start the MRP recovery process, and so on after the recovery is complete, re-open the ADG real-time application, all normal.

shutdown immediatestartup mountalter database recover managed standby database disconnect from session;此时等待恢复完成...alter database recover managed standby database cancel;alter database open;alter database recover managed standby database using current logfile disconnect from session;

Check the standby status confirm everything is OK:

Sql> SELECT Open_mode, Database_role, Switchover_status, force_logging, Dataguard_broker, GUARD_STATUS from V$ DATABASE; Open_mode database_role switchover_status for Dataguar guard_s----------------------------------------- ---------------------------------READ only with the APPLY physical STANDBY not allowed YES DISABLED nonesql> SE                                                            Lect * from V$dataguard_stats;name VALUE UNIT time_computed datum_time------------------------------------------- ----------------------------------------------------- ------------------------------ ----------------------------                                                     --------------------------------transport Lag +00 00:00:00                        Day (2) to second (0) interval 01/17/2017 17:42:26 01/17/2017 17:42:26apply Lag                   +00 00:00:00                                  Day (2) to second (0) interval 01/17/2017 17:42:26 01/17/2017 17:42:26apply F   Inish Time +00 00:00:00.000 Day (2) to second (3) interval 01/17/2017 17:42:26estimated Startup Time Secon D 01/17/2017 17:42:26

5. Check MoS to locate the root cause
Query MoS found this behavior matches the bug 10419984
Bug 10419984:active DATA GUARD STANDBY GIVES ORA-600 [Kcbr_apply_change_11]
It is recommended that the patch be applied to prevent the issue from being triggered again.

Oracle 11.2.0.1 ADG Environment MRP process encounters Ora