ORA-21561, ORA-15055, ORA-25253 lead to DG slave database cannot apply archive, ora-21561ora-15055

ORA-21561, ORA-15055, ORA-25253 lead to DG slave database cannot apply archive, ora-21561ora-15055
Yesterday I went to a customer to do inspection, by the way, to see whether the last build of the RAC-DG environment is normal, do not see do not know, a look at the fright, the last DG was run on August 20, the instance of the DG slave database has not been started since January 1, August 31. After inquiry, it was learned that the power was cut off once that day, and then the machine was restarted. I started the instance until today. That is to say, from January 1, August 31 to today, the Standby database is in the unused state.
Next, check the backup database archive. The last log of tnread1 is 1661, And the last log of tnread2 is 1324, at this time, the earliest logs retained in the master database were in September 8. thread 1 was 2055 at the earliest and thread 2 was 1555 at the earliest. There are hundreds of archive errors between the master and slave databases (normal, it's almost a month before the instance is started)
ASMCMD> release/2014_09_09/2014_09_10/release/2014_09_19/2014_09_20/2014_09_21/release cd 2014_09_08ASMCMD> quit (omitted) ......
Round (Omitted )...... ASMCMD>
Although the archive is deleted using delete input after the archive is backed up in the script to reduce the disk space occupied by the archive, you can see in the backup log of the RMAN script that from January 1, August 31, one after another reported RMAN-08137, prompt because the slave database has not been archived, resulting in the failure to delete:
Archive log file name = + DATA/sis/archivelog/2014_08_31/thread_eclipseq_1661.1208.85701_81 RECID = 3930 STAMP = 857020.81rman-08133: warning: the archive log has not been deleted, because the backup or upstream capture process requires that it archive the log file name = + DATA/sis/archivelog/2014_08_31/thread_eclipseq_1662.1212.857042297 thread = 1 Series = 1662rman-08133, because the backup or upstream capture process requires it to archive the log file name = + DATA/sis/archivelog/2014_09_01/thread_2_seq_1325.1204.857122335 thread = 2 series = 1325rman-08133: warning: the archive log is, because the standby or upstream capture process needs it
However, since the FRA disk space is limited, Oracle will automatically clear the content to release space when a certain percentage is used (with parameters adjustable, therefore, archive logs in FRA are retained for about 18 days, from October 25 to October 25, and archive logs in December are definitely unavailable. The most recent backup set is only available in December 25.
So I decided to re-build the DG, shut down the slave database instance, delete all database files (data files, control files, log files), and only keep the password files, parameter files, and tnsnames. ora, listener. ora is enough, and reconstruction is very convenient. Use the 11g duplicate to re-synchronize it. The command is as follows:
Rman target/auxiliary sys/oracle @ sisdgRMAN> run {
Allocate channel c1 device type disk; allocate auxiliary channel c2 device type disk; set newname for tempfile 1 to 'd: \ app \ administrator \ oradata \ sis \ temp.269.852648395 '; duplicate target database for standby from active database dorecover; release channel c1; release channel c2 ;}
After the above operations are performed, the archive between the slave database and the master database is synchronized.
Master database:
SQL> archive log list database log mode archive mode automatic archiving enable archiving end point USE_DB_RECOVERY_FILE_DEST oldest online log sequence 2617 next archive log sequence 2619 current log sequence 2619SQL> select thread #, max (sequence #) from v $ archived_log group by thread #;
THREAD # MAX (SEQUENCE #) ---------- -------------- 1 2619 2 2556
Slave database:
SQL> archive log list database log mode automatic archiving mode enable archiving end point D: \ archivelog earliest online log sequence 0 next archived log sequence 0 current log sequence 0SQL> select thread #, max (sequence #) from v $ archived_log group by thread #;
THREAD # MAX (SEQUENCE #) ---------- -------------- 1 2619 2 2556
The master and slave databases check v $ archive_dest_status. The status columns on both sides are valid. Therefore, the redo apply feature of the slave database is enabled and logs are applied.
SQL> select thread #, sequence #, applied from v $ archived_log;
THREAD # SEQUENCE # APPLIED ---------- ---------------- 1 2617 YES 1 2618 YES 2 2556 YES 1 2619 NO 1 2620 NO
Because the lgwr async mode is used to transmit logs, the standby redo logfile is re-created. Each thread in the master database has three groups of logs, so the slave database has created seven groups of logs.
In addition, an error is reported in the alertlog of the standby database because it is from the primary database duplicate, and the configuration information of RMAN retains some parameters of the primary database:
Starting control autobackupGot error: 19624 ******************** WARNING ****************** * ******** The errors during Server autobackup are not fatal, as itis attempted after sucessful completion of the command. however, it is recomended to take an RMAN control filebackup as soon as possible because the Autobackup failedwith the following error: ORA-19624: operation failed, retry possibleORA-19504: fa Iled to create file "C: \ ORABACKUP \ BACKUPSETS \ SIS1-C-3160648191-20140925-02.CTL" ORA-27040: file create error, unable to create fileOSD-04002: unable to open file O/S-Error: (OS 3) the specified path cannot be found. * ****************** End of warning ***************** **
RMAN> show all;

Use the target database control file TO replace the RMAN Configuration Parameter of the database with the recovery directory db_unique_name SISDG: configure retention policy to redundancy 7; configure backup optimization off; # defaultCONFIGURE default device type to disk; configure controlfile autobackup on; configure controlfile autobackup format for device type disk to 'C: \ orabackup \ backupsets \ sis1-% F. ctl '; configure device type disk parallelism 1 backup type to backupset; configure datafile backup copies for device type disk to 1; # defaultCONFIGURE archivelog backup copies for device type disk to 1; # defaultCONFIGURE maxsetsize to unlimited; # default
Configure encryption for database off; # defaultCONFIGURE ENCRYPTION comment 'aes128 '; # defaconfigconfigure COMPRESSION comment 'basic' as of release 'default' optimize for load true; # defaultCONFIGURE archivelog deletion policy to none; # defaultCONFIGURE snapshot controlfile name to 'd: \ APP \ ADMINISTRATOR \ PRODUCT \ 11.2.0 \ DBHOME_1 \ DATABASE \ SNCFSIS. ORA '; # default
The standby database does not have the path "'C: \ orabackup \ backupsets \". Therefore, if the path is clear, RMAN> configure controlfile autobackup format for device type disk clear is returned;

The problem seems to have been solved successfully, but this is not the case. After I switch logs several times in node 1, I found that although the archive can be smoothly uploaded to the slave database, however, the subsequent logs will never be applied. You can view Node 1 and the standby database's alertlog, and no exceptions are found until I realized that I should check node 2, just to find a clue.
Normally, the status of the remote archiving path queried in the v $ archive_dest_status view must be valid and the error column is empty, the query result of Node 2 is as follows:
SQL> select dest_id, status, error from v $ archive_dest where dest_id <3;
DEST_ID status error ---------- ------------------------------------ 1 VALID 2 ERROR ORA-21561: generating OID failed
Now go to view Node 2 alertlog, found in addition to reporting ORA-21561, along with the ORA-15055, ORA-25253, etc.
Thu Sep 25 15:05:06 2014 WARNING: ASM communication error: op 0 state 0x0 (15055) ERROR: direct connection failure with ASMWARNING: ASM communication error: op 0 state 0x0 (15055) ERROR: direct connection failure with ASMWARNING: ASM communication error: op 0 state 0x0 (15055) ERROR: direct connection failure with ASMWARNING: ASM communication error: op 0 state 0x0 (15055) ERROR: direct connection failure with ASMWARNING: ASM communication error: op 0 state 0x0 (15055) ERROR: direct connection failure with ASMThu Sep 25 15:05:37 2014 Error 21561 lost Ed logging on to the standbySuppressing further error logging of protected Sep 25 15:05:46 2014ORA-25253 encountered when generating server alert SMG-3503Thu Sep 25 15:05:58 2014 Thread 2 advanced log sequence 2561 (LGWR switch) current log #6 seq #2561 mem #0: + DATA/tc/onlinelog/group_6.1523.854793175 Current log #6 seq #2561 mem #1: + DATA/tc/onlinelog/group_6.1524.854793175Thu Sep 25 15:05:58 2014 Archived Log entry 6148 added for thread 2 sequence 2560 ID 0xbc63a5fc dest 1: thu Sep 25 15:05:58 2014 Error 21561 caught ed logging on to the standbySuppressing further error logging of LOG_ARCHIVE_DEST_2.FAL [server, ARC3]: FAL archive failed, see trace file. ARCH: FAL archive failed. archiver continuingThu Sep 25 15:07:49 2014ORA-25253 encountered when generating server alert SMG-3503Thu Sep 25 15:08:07 2014 ARCH: Possible network disconnect with primary databaseThu Sep 25 15:11:38 2014 Error 21561 lost Ed logging on to the standby
Execute lsnrctl status on node 2 to check the listening status and use tnsping net service name. hang is there and no result is displayed.
So I checked the documents about ORA-21561 and ORA-15055 on MOS, there are several related documents have mentioned this phenomenon, the main description is as follows: 1. usually occurs after 10.2.0.x upgrade. When you log on to SQLPLUS, you will be prompted: ORA-21561: oid generation failed... /Add the complete host name of localhost to hosts. I can log on to SQLPLUS normally. It seems that this is not the case I encountered.

2. Caused by a bug, Bug 14324057 and Bug 12529945 may potentially consume desktop heap memory. You need to modify the Registry to increase the default desktop heap size.
The document also mentions that the bug is targeted at 11.2.0.2 patch 19--11.2.0.3 patch 7, but the library has already hit patch 27. If it is a bug, it should have been fixed. So I tried to modify the registry and modified the SharedSection under \ HKEY_LOCAL_MACHINE \ SYSTEM \ CurrentControlSet \ Control \ Session Manager \ SubSystems \ 1024,. Back up the registry before modification.

Before modification:
% SystemRoot % \ system32 \ csrss.exe ObjectDirectory = \ Windows SharedSection = 768, 20hrs, Windows = On SubSystemType = Windows ServerDll = basesrv, 1 ServerDll = winsrv: UserServerDllInitialization, 3 ServerDll = winsrv: success, 2 ServerDll = sxssrv, 4 ProfileControl = Off MaxRequestThreads = 16
After modification:
% SystemRoot % \ system32 \ csrss.exe ObjectDirectory = \ Windows SharedSection = 1024, 20hrs, Windows = On SubSystemType = Windows ServerDll = basesrv, 1 ServerDll = winsrv: UserServerDllInitialization, 3 ServerDll = winsrv: success, 2 ServerDll = sxssrv, 4 ProfileControl = Off MaxRequestThreads = 16
Note: After the registry is changed, you need to restart the node host before it takes effect.