Some time ago with a set of 11203 rac adg on EXADATA Machine environment, in the ADG standby side node2 through dblink query prompt ora-1089 error, but in node1 query normal, the recover process of DG is on node1. then it is confirmed that it is a simple bug record.
-- On standby node2, but on standby node1 was worked
Select count (*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
ORA-01089: immediate shutdown in progress-no operations are permitted
Process ID: 771
The Session ID is 1517 Serial number: 4211.
-- Diag
Alter session set events '2014 trace name context forever, level 12: 10046 trace name errorstack level 3 ';
Select count (*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
-- Error
Call Stacke in trace file
----- Call Stack Trace -----
Calling call entry argument values in hex
Location type point (? Means dubious value)
----------------------------------------------------------------------------
Skdstdst () + 36 call kgdsdst () 000000000? 000000000?
7FFFB26EA518? 000000001?
000000001? 000000002?
Ksedst1 () + 98 call skdstdst () 000000000? 000000000?
7FFFB26EA518? 000000001?
000000000? 000000002?
Ksedst () + 34 call ksedst1 () 000000000? 000000001?
7FFFB26EA518? 000000001?
000000000? 000000002?
DbkedDefDump () + 2741 call ksedst () 000000000? 000000001?
7FFFB26EA518? 000000001?
000000000? 000000002?
Ksedmp () + 36 call dbkedDefDump () 000000003? 000000000?
7FFFB26EA518? 000000001?
000000000? 000000002?
DbkdaKsdActDriver () calling ksedmp () 000000003? 000000000?
+ 1960 7FFFB26EA518? 000000001?
000000000? 000000002?
DbgdaExecuteAction (call dbkdaKsdActDriver () 7f86b43000010? 7FFFB26F11A0?
) + 1065 7FFFB26EA518? 000000001?
000000000? 000000002?
DbgdaRunAction () + 81 call dbgdaExecuteAction (7f86b43000010? 00A1B8DA0?
0) 0020C0003? 7FFFB26F11A0?
000000001? 000000002?
DbgdRunActions () + 59 call dbgdaRunAction () 7f86b43000010? 000000005?
0020C0003? 7FFFB26F11A0?
000000001? 000000002?
DbgdProcessEventAct call dbgdRunActions () 7f86b43000010? 000000005?
Ions () + 651 0020C0003? 7FFFB26F11A0?
000000001? 000000002?
DbgdChkEventKgErr () call dbgdProcessEventAct 7f86b43000010? 00BC1D9A0?
In MOS, the Bug 17162712 is consistent. This bug affects 11.2.0.3/4. The bug was modified only in 12.2 (Future Release) 12.1.0.2 (Server Patch Set; you can also provide ONE patch for minor fixes in versions not before 12C, or restart standby db to bypass this bug, as shown below:
This bug is only relevant when using Real Application Clusters (RAC) and Database Link/Distributed
If the recovering instance of a RAC standby fails, one of the remaining instances will do special instance recovery.
After this recovery one of the shutdown flags is not cleared, triggering the ORA-1089: "immediate shutdown in progress"
When a database link is used.
During the recovery an ORA-1089 is normal, the problem is that the error is raised even after the recovery
Stack trace:
----
Skdstdst <-ksedst1 <-ksedst <-dbkedDefDump <-ksedmp <-dbkdaKsdActDriver
<-DbgdaExecuteAction <-dbgdaRunAction <-dbgdRunActions <-
DbgdProcessEventActions <-dbgdChkEventKgErr <-dbkdChkEventRdbmsEr <-ksfpec
<-DbgePostErrorKGE <-1129 <-dbkePostKGE_kgsf <-kgeselv <-ksesecl0 <-
K2gInsert <-k2lbeg <-k2sibg <-npibeg <-kpnpre <-upirtrc <-kpurcsc <-
Kpuexec <-OCIStmtExecute <-OCIKGetDescInfo <-ddfnetCFull <-ddfnet2Normal
<-Kkmfcbrm <-kkmpfcbk <-qcsprfro <-qcsprfro_tree <-
Qcspafq <-qcspqbDescendents <-qcspqb <-kkmdrv <-opiSem <-opiprs <-
KksParseChildCursor <-rpiswu2 <-kksLoadChild <-kxsGetRuntimeLock <-kksfbc
<-Kkspsc0 <-kksParseCursor <-opiosq0 <-kpooprx <-kpoal8 <-opiodr <-
Ttcpip <-opitsk <-opiino <-opiodr <-opidrv <-sou2o <-opimai_real <-
Ssthrdmain <-main <-libc_start_main <-start
# Restart STANDBY INSTANCE
[Oracle @ qdexa1db01 (orarpt1) oracle] $ srvctl config database
Rptstby
[Oracle @ qdexa1db01 (orarpt1) oracle] $ srvctl stop database-d rptstby
[Oracle @ qdexa1db01 (orarpt1) oracle] $ srvctl start database-d rptstby
SQL> select name, open_mode, database_role from gv $ database;
NAME OPEN_MODE DATABASE_ROLE
---------------------------------------------
ORARPT MOUNTED PHYSICAL STANDBY
ORARPT MOUNTED PHYSICAL STANDBY
# Open all standby instances
SQL> alter database open;
SQL> select name, open_mode, database_role from gv $ database;
NAME OPEN_MODE DATABASE_ROLE
---------------------------------------------
ORARPT READ ONLY PHYSICAL STANDBY
ORARPT READ ONLY PHYSICAL STANDBY
SQL> alter database recover managed standby database PARALLEL 48 using current logfile disconnect from session;
Database altered.
# In both instances run same queries over dblink not with ora-1089 error also Now.
SQL> select count (*) from dbmt.dbmt_tableinfo@LINK_WEEJAR_A2.ANBOB.COM;
COUNT (*)
----------
317