Common RMAN problems in 11GR2

This article is a summary of Oracle Support's problems in the 11GR2RMAN backup process. Common RMAN problems in 11GR2: A few structural changes in 11gR2 have a wide impact on RMAN settings. the SnapshotBackup (Snapshot Backup) control file location must be in the sharing location in the RAC environment. In 11gR2 and later versions

This article is a summary of Oracle support's problems in the 11GR2 RMAN backup process. Common RMAN problems in 11GR2: A few structural changes in 11gR2 have a wide impact on RMAN settings. the Snapshot/Backup control file location must be in the sharing location in the RAC environment. In 11gR2 and later versions

This article is a summary of Oracle support's problems during the 11GR2 RMAN backup process.

Common RMAN problems in 11GR2

A few schema changes in 11gR2 have a wide impact on RMAN settings

1. The Snapshot/Backup control file location must be in the sharing location in the RAC environment.

In 11gR2 and later versions, the backup of the control file does not hold CF enqueue during execution. This does not affect non-RAC databases. However, for the RAC database, any instance in the cluster can be written to the snapshot/backup control file because the file backup mechanism is changed in 11gR2. Therefore, the Snapshot control file must be visible to all instances. This also exists when you directly create a backup of the control file from SQL * Plus. Any instance in the cluster can be written to the backup control file. Control file backup. Even if you use SQL "alter database backup controlfile...", you must create a backup on the shared device.

The Snapshot control file must be accessible to all nodes in the RAC database. If the snapshot control file is not on the shared device, the snapshot of the control file is obtained from the RMAN backup) will cause an error. This applies to the use of SQL * Plus backup control files and the configuration of automatic backup of control files in non-shared locations.

RMAN-00571: ========================================================== ======================
RMAN-00569: ============== error message stack follows ==================
RMAN-00571: ========================================================== ========================
RMAN-03009: failure of resync command on default channel at 03/13/2012 10:19:41
ORA-00245: control file backup operation failed

The solution is to change the Snapshot/backup (Snapshot/backup) control file location to the shared device, otherwise it will fail with the ORA-245: control file backup operation failed.

Note 1472171.1 In RAC environment from 11.2 onwards Backup Or Snapshot controlfile needs to be in shared location is released for this issue.

2. The MMON process automatically backs up control files after the structure changes.

In the release earlier than 11GR2, each DDL command for changing the database structure creates an automatic backup of the control file. You can see in alert. log that after each DDL command is executed, there will be a message about automatic backup and creation of the control file. When multiple structural changes are made at the same time, this may cause serious performance problems.

The Controlfile Autobackup Deferral function has been implemented since Oracle Database 11g Release 2. When the application script contains multiple structural changes (for example, adding a tablespace, changing the tablespace or data file status, adding a new online redo log, and renaming a file ), RMAN performs automatic backup of the control file only once. In 11gR2, the control file is automatically backed up by the MMON slave process within several minutes after the structure change, which improves performance. Therefore, it is normal to obtain the automatic backup of control files after several minutes of structural changes to the database. At the same time, it is normal that the message about automatic backup of control files is not displayed in alert. log.

The MMON subordinate process responsible for backing up the control file will generate a tracking file containing the information required to create the control file and name it: _ M000 _ . Trc

3. You can perform recovery zone backup on the disk.

Before 11gR2, you can only back up the Flash Recovery Area (FRA) on the tape. From 11GR2, FRA backup can be performed on the disk. The "to destination" clause is added TO the BACKUP command. This option can specify a specific directory location for BACKUP to the disk. This option is mainly used for the backup recovery area command. If backup optimization is enabled, RMAN skips backups of the same files in the directory specified by the "to destination" clause.

RMAN> Backup recovery area to destination '+ data ';

Common Bugs affecting RMAN stability in 11GR2.

1. BUG 9877980: SR11.2.0.2CSHAR _ FORCE-TRC-KKSLMARKLITERALBINDS

Symptoms:

Database Version: 11.2.0.2, CURSOR_SHARING is FORCE or SIMILAR

RMAN backup failed with ORA-01008 or various errors

DBGSQL: TARGET> select count (*) into: dbstate from v $ parameter where lower (name) = '_ dummy_instance' and upper (value) = 'true'
DBGSQL: sqlcode = 1008
RMAN-00571: ========================================================== ========================
RMAN-00569: ============== error message stack follows ======================
RMAN-00571: ========================================================== ========================
ORA-01008: not all variables bound

Or

RMAN-00571: ========================================================== ==================
RMAN-00569: ============= error message stack follows ==================
RMAN-00571: ========================================================== ==================
RMAN-03002: failure of allocate command at 00:38:14
ORA-03114: not connected to ORACLE

And alert. log shows

ORA-07445: exception encountered: core dump [kkslMarkLiteralBinds () + 240] [SIGBUS] [ADDR: 0x18222d02020d29] [PC: 0x1049E96D0] [Invalid address alignment] []
Or
ORA-07445: exception encountered: core dump [kkslpli () + 212] [SIGSEGV] [ADDR: 0xFFFFFFFF7A973288] [PC: 0x1049FEE14] [Invalid permissions for mapped object] []

The root cause is bugs 9877980. This Bug is fixed in 11.2.0.3. This Bug has one-off patch available.

Workaround: clears the Shared Pool.

Ref:

Bug 9877980-ORA-7445 [kkslMarkLiteralBinds]/Assorted Errors on 11.2.0.2 if cursor sharing is enabled Note 9877980.8

Alert: Note 1472116.1 RMAN backup fails with Assorted Errors on 11.2.0.2 if cursor sharing is enabled

2. Bug 8482162: SNAPSHOT CONTROLFILE STILL NEED NOAC ON NFS MOUNT POINT

If the control file automatic backup target file system is NFS, the file system is required to be mounted using the NOAC option; otherwise the ORA-27054 will be reported

Symptoms:

If the snapshot (snapshot) control file is located to the NFS file system and is not mounted using the option "NOAC", The RMAN backup fails with a ORA-27054 error. According to Bug 5064356, if 10.2.0.4.0 or later is running, the NFS mount point NOAC option is no longer needed. However, this fix only appears to apply to specific file types, such as online redo logs, archived redo logs, backup slices (such as RMAN), and trace files.

Starting Control File and SPFILE Autobackup at 07-APR-12
RMAN-571: ========================================================== ==============================
RMAN-569: ==================== error message stack follows ==========================
RMAN-571: ========================================================== ==============================
RMAN-3009: failure of Control File and SPFILE Autobackup command on
ORA_DISK_1 channel at 04/07/2012 10:29:17
ORA-1580: error creating control backup file
/Oracle/product/10.2.0/db_1/dbs/snapcf_COPA1.f
ORA-27054: NFS file system where the file is created or resides is not
Mounted with correct options
Additional information: 3

In the alert. log File

Starting control autobackup
WARNING: NFS file system/oracle/product mounted with incorrect options
WARNING: Expected NFS mount options:
Rsize> = 32768, wsize> = 32768, hard, actimeo = 0
Autobackup failed with following error
ORA-1580: error creating control backup file
/Oracle/product/10.2.0/db_1/dbs/snapcf_COPA1.f
ORA-27054: NFS file system where the file is created or resides is not mounted with correct options
Additional information: 3


This problem occurs because the Bug 8482162: snapshot controlfile still need noac on nfs mount point has been fixed in 11.2.0.2.

Workaround:

1) Use the NOAC option to mount the NFS file system that saves the snapshot control file.

Or

2) change the position of the snapshot control file to non-NFS.

Ref: Bug 8482162 Snapshot controlfile still needs "noac" on NFS mount points (ORA-27054) Note 8482162.8

3. Bug 9577583: FALSE ORA-942 OR OTHER ERRORS WITH MULTIPLE SCHEMAS HAVING IDENTICAL OBJECTS

When the RMAN directory database (catalog) saves multiple RMAN directory Schemas (scheme), the directory database will remind you of an error ORA-00942.

Symptoms:
Multiple ORA-942 errors discovered by the user
This problem may occur in any database with objects of the same name under multiple Schemas (schemes.
This is an intermittent problem that cannot be reproduced, but may occur multiple times.
This is a common Bug that mainly affects RMAN directory backup. Affected 11.2.0.1. fixed in 11.2.0.2.

RMAN-00571: ========================================================== ==============================
RMAN-00569: ==================== error message stack follows ==========================
RMAN-00571: ========================================================== ==============================
RMAN-03015: error occurred in stored script incremental_level0
RMAN-03002: failure of backup command at 06/25/2010 13:21:22
RMAN-06004: ORACLE error from recovery catalog database: ORA-00942: table or view does not exist
RMAN-06031: cocould not translate database keyword
Debug tracing information display

DBGSQL: exec SQL AT RCVCAT begin dbms_rcvman. translateDatabase; end; [13:21:22. 794]
DBGSQL: sqlcode =-942 [13:21:22. 795]
DBGMISC: krmksqlerror called from file krmk. c, line 12649 [13:21:22. 796]
DBGRCVMAN: ENTERING translateDatabase
DBGRCVMAN: ENTERING validateState
DBGRCVMAN: EXITING validateState
DBGRCVMAN: OPENING cursor translateDatabase_c in translateDatabase
DBGMISC: krmicomp: error 6004 signalled during compilation [13:21:22. 797]
DBGMISC: ENTERED krmkmrsr [13:21:22. 797]

RMAN-06004: ORACLE error from recovery catalog database: ORA-00942: table or view does not exist
ORA-06512: at "CALYPP. DBMS_RCVMAN", line 4590
ORA-06512: at "CALYPP. DBMS_RCVMAN", line 16168
ORA-06512: at line 1
RMAN-06097: text of failing SQL statement: begin dbms_rcvman. translateDatabase; end;
RMAN-06099: error occurred in source file: krmk. pc, line: 7618
RMAN-06031: cocould not translate database keyword

Solution:

Application Patch 9577583 for Bug 9577583

Ref: Note 9577583.8 False ORA-942 or other errors when multiple schemas have identical object names.

4. BUG 10635701-backup of fra consumes 15 gb of pga and fail with ORA-4030

When backing up a large number of files, RMAN fails with a ORA-4030 due to excessive memory consumption. The error occurs in heap (heap) KSFQ, which contains the block with the comment "KSFQ Buffers. Affected 11.2.0.1 and 11.2.0.2, which have been fixed in 11.2.0.3
Symptoms:

The RMAN trace information shows errors in the following functions.

Dbms_backup_restore.validationvalidate, with a trace line similar to the following:
Krmxrpc-channel t1 kpurpc2 err = 4030 db = target proc = SYS. DBMS_BACKUP_RESTORE.VALIDATIONVALIDATE

Call Stack of the failed process:

Kghalf <-ksfqbalo <-ksfqbcre<-ksfqxc <-ksfqxcrx <-ksfqxre <-krbrvv

The allocated memory is the heap (heap) of KSFQ. The KSFQ heap (heap) contains the block with the comment "KSFQ Buffers. This information is dumped to a trace file generated by the error ORA-4030

Errors in the following files:/cihcissdb028/dump01/oracle/dxls/udump/diag/rdbms/dxls/incident/incdir_68404/dxls_ora_27140_i68404.trc:

ORA-04030: out of process memory when trying to allocate 824492 bytes (pga heap, kco buffer)
ORA-04030: out of process memory when trying to allocate 1052684 bytes (KSFQ heap, KSFQ Buffers)

Solution:

Apply Patch 10635701, which cannot be bypassed. Affected 11.2.0.1 and 11.2.0.2, which have been fixed in 11.2.0.3.

Ref: Note 10635701.8 RMAN backup temporary files consumes lots of PGA/fails with ORA-4030



5. BUG 12370722-CATUPGRD. SQL HANGS WITH ARCH ERROR ORA-600 [KRR_HIGHEST_SCN_TIM_8]

After upgrading to 11.2.0.2, the archiving process continuously triggers ORA_0600 [krr_highest_scn_tim_8]. An error is reported in 11.2.0.2 after the upgrade, which affects the upgrade and causes downtime. The solution is to clear online redo logs. This issue has been fixed in 11.2.0.3.

The bugs listed below are similar to the parent Bug 12370722.

Bug 11872889: ORA-600: internal error code, ARGUMENTS: [KRR_HIGHEST_SCN_TIM_8]

Bug 12534566: database open failed ORA-00600 [KRR_HIGHEST_SCN_TIM_8]

Bug 11062394: ORA-600 [KRR_HIGHEST_SCN_TIM_8] REPORTED BY AN ARCHIVER PROCESS

All these bugs have been closed and are repeated with the following bugs:

Bug 12370722: CATUPGRD. SQL HANGS WITH ARCH ERROR ORA-600 [KRR_HIGHEST_SCN_TIM_8]

Symptoms:

Search error:
? Run Oracle version 11.2.0.2
? The DATABASE was recently upgraded from 10.2 (or 10.1) to 11.2.0.2. To confirm this, 11.2.0.2 alert log should display "alter database open migrate ".

The archive process periodically (for example per minute) reports an error ORA-00600: [krr_highest_scn_tim_8] And then terminates, the call stack is as follows:
-> Ksbrdp-> krsv_abs-> ksbabs-> kcrrwk-> kcrrwkx-> krse_arc_driver-> kernel-> krse_arc_spool-> kernel-> ora600: [kernel]

Or

The process of trying to open the database reports an error ORA-00600: [krr_highest_scn_tim_8], the call stack is as follows:
-> Adbdrv-> kcfopd-> kcttsc-> krsq_arch_to_force _-> krse_arc_driver-> role-> krse_arc_spool-> krr_highest_scn_tim-> ora600: [Response]

Or

The process that executes alter system archive log all; reports an error ORA-00600: [krr_highest_scn_tim_8], the call stack is as follows:

-> Kkyasy-<strong-> krsq_arch_one_log-> krse_arc_driver-> krse_arc_driver_core-> krse_arc_spool-> krr_highest_scn_tim-> ora600: [strong]
A specific online redo log is not archived. The following query displays the Unarchived log serial number:

SQL> select sequence # from v $ log where archived = 'no' and status = 'current ';

Error:

RMAN-00571: ========================================================== ====================
RMAN-00569: ============= error message stack follows ================
RMAN-00571: ========================================================== ====================
RMAN-03009: failure of SQL command on default channel at 05/20/2011 01:26:24
RMAN-11003: failure during parse/execution of SQL statement: alter system archive log current
ORA-00600: internal error code, arguments: [krr_highest_scn_tim_8], [], [], [], [], [], [], [], [] []

Workaround:

To prevent ORA-00600: [krr_highest_scn_tim_8], do not return and open the database with Oracle version 10 after you start upgrading to 11.2.0.2.

If the database is suspended because it cannot switch to the next (Unarchived) Online redo log, or because the foreground process attempts to archive the online redo log and the ORA-00600: [krr_highest_scn_tim_8] error occurs, if the database cannot be opened, add another redo log group.

SQL> startup mount

Alter database add logfile group (' ') Size M;

If the error ORA-00600 has been reported: [krr_highest_scn_tim_8], and regular Continuous Error Reporting, you can solve it by one of the following methods:

-Install the patch,

-Alternatively, clear online redo logs using the following methods:

SQL> select group # from v $ log

Where archived = 'no' and status = 'current ';

Alter database clear logfile group ;

Note: clearing the online redo log indicates that the redo in the log of this serial number cannot be used for restoration. Therefore, you should perform the full backup as early as possible after clearing the log.

6. BUG 10170431-CTWR CONSUMING LOTS OF CPU CYCLES

If Block Change Tracking (BCT) is enabled, the CTWR process consumes CPU, and the overall database performance will be seriously affected. This happens in 11.2.0.2. The solution is to disable block change tracking or apply the one-off patch. This problem has been fixed in 11.2.0.3.

Symptoms:

At the lowest load, the CTWR background process consumes 90% to 100% of the CPU.

Alter system checkpoint significantly reduces the ctwr cpu load and will return 90% to 100% CPU load later (after CTWR processes block changes.

The call stack of CTWR is likely to be displayed in the following function in a continuous loop (spinning ):

Kcmgtsf ()
Krcptmo ()
Ksbabs ()
Krcpabs ()
Ksbrdp ()



Workaround: Disable block change tracking (BCT) or apply the Patch 10170431 for bug 10170431.



7. BUG 13000553-rman backup fails with RMAN-20999 ERROR AT STANDBY DATABASE

RMAN backup or resync RMAN directory (resync catalog) command failed with the error RMAN-20052: invalid datafile create SCN

Add the data file to the transportable tablespace and restore the directory.

Since the insertion (plug in) SCN is zero, a problem occurs when you try to restore the directory.

The solution is to apply patch 13000553.

Bug 13877582-RMAN SOME DATAFILES AT STANDBY SITE APPEAR WITH NULL NAME

It is found to be the same as the following Bug:

Bug 13000553-rman backup fails with RMAN-20999 ERROR AT STANDBY DATABASE

Symptoms:
The target database is a physical standby uard environment.
The tablespace has been inserted into the primary database.
After the plug in tablespace is inserted, some data files are added to it.
Restore the directory to 11.2.0.3

Fixed in versions earlier than 12.1

Solution

Apply patch 13000553 in the recovery directory database and re-Synchronize the directory between the master site and the backup site. If the file name is still blank after the patch is applied, re-create the recovery directory, register the master site in the new directory, and then re-Synchronize the backup site with the new directory.

Ref:

Note 1446934.1 Some Datafiles At Standby Site Appear With NULL Name in RMAN

Note 1411883.1 RMAN-20052: invalid datafile create SCN During Resync or Backup using Recovery Catalog 11.2.0.3



8. BUG 12312133-RMAN INCREMENTAL BACKUP WITH BLOCK CHANGE TRACKING MAY CAUSE STANDBY DB CRASH

If BCT is enabled on the standby database and RMAN performs an incremental backup, CTWR causes the standby database to have a ORA-0600 [krcccb_busy] and crash. This issue affects 11.2.0.2 and 11.2.0.3. You can disable block change tracking by bypassing the issue.

Symptoms:
BCT enabled on the standby Database
RMAN performs Incremental backup.
CTWR displays the ora-0600 [krcccb_busy], causing the standby database to crash.

Errors in file/u01/app/oracle/diag/rdbms/mbcdwps/MBCDWPS1/trace/MBCDWPS1_ctwr_499736.trc:

ORA-00600: internal error code, arguments: [krcccb_busy], [], [], [], [], [], [], [], [], [], []

CTWR (ospid: 499736): terminating the instance due to error 487

System state dump requested by (instance = 1, osid = 499736 (CTWR), summary = [abnormal instance termination].

System State dumped to trace file/u01/app/oracle/diag/rdbms/mbcdwps/MBCDWPS1/trace/MBCDWPS1_diag_1884206.trc

Workaround: solution: Disable block change tracking. Application patch 12312133

Ref: Note 12312133.8 Standby DB crashes with ORA-600 [krcccb_busy]/Ora-00600 [krccckp_scn] with block change tracking

9. BUG 10318078-rman coredumps on exit: BACKUP (to tape) COMPLETES SUCCESFULLY

In MySQL 11.2, the core dump is generated when RMAN is successfully backed up to the tape and RMAN is exited.

Symptoms:
Backup is successfully completed. When RMAN exits, core dump is generated ).
Core stack:/oracle/movt/11G/app/product/release/lib/libskgxp11.so

Workaround: link the Oracle executable file to the static version of the Media Manager API library, instead of the dynamic link library.

Close all instances that use this ORACLE_HOME

% Cd $ ORACLE_HOME/rdbms/lib

% Make-f ins_rdbms.mk ioracle LLIBMM =/usr/lib/libnwora.

% Ln-s/usr/lib/libnwora. so libobk. so

Use the static Link Library "libnwora. a" instead of the dynamic library "libnwora. so"

Ref: Note 1296704.1 rman coredumps on exit: BACKUP (to tape) completes succesfully.

Solution:

Bug fixes for applications: 10318078

Patch 11774404: tracking bug for 10318078 FOR AIX11.2-212-ibm aix on POWER Systems (64-bit)

Patch 11774413: tracking bug for 10318078 for solaris SPARC64



10. BUG 13602883-BLOCK CHANGE TRACKING NOT USED FOR INCREMENTAL BACKUPS

Symptoms:

If the block change tracking (BCT), RMAN session, or instance for Incremental backup is accidentally disabled during database (cluster_database = TRUE) running, and the RDBMS release is earlier than 11.2.0.4, this Bug may be hit.

This Bug affects 11.2.0.2/3 (or earlier versions), and may occur on any platform. However, it only affects RAC, that is, the database has set the parameter cluster_database = true.

This Bug has been fixed in 11.2.0.4 and later versions.



11. MML incompatibility issues: ORA-07445 [Strcpy () + 48] occurs during an RMAN backup or recovery to tape operation using Oracle 11.2

Incompatible MML software will cause core dump when using RMAN to back up the database. The core dump error ORA-07445 [Strcpy ()] is reported in alert. log and the backup channel is unexpectedly terminated.

Symptoms:

Errors in file/apps/oh/admin/PPMP/bdump/diag/rdbms/ppmp/PPMP/trace/PPMP_ora_32421.trc (incident = 29135 ):
ORA-07445: exception encountered: core dump [strcpy () + 16] [SIGSEGV] [ADDR: 0x9] [PC: 0x3E2B170D00] [Address not mapped to object] []

RMAN-00571: ========================================================== ====================
RMAN-00569: ============= error message stack follows ================
RMAN-00571: ========================================================== ====================
RMAN-03009: failure of backup command on t1 channel at 11/18/2011 23:13:19
RMAN-10038: database session for channel t1 terminated unexpectedly

Call Stack ):
_ Restore_rt strcpy put_string send_bprdreq bprd_get_features when using int_StartJob sbtbackup skgfqcreksfq_re ksf1_crx krbbOpenOutput krbbpc krbibpc

Ref: Note 959015.1 ORA-07445 [Strcpy () + 48] during RMAN backup or restore to tape using Oracle 11.2

Solution:

Check MML software and 11.2 compatibility. If you need help, contact the supplier technical support.

For example, you can find details related to Veritas netbackup at the following URL.

Http://www.symantec.com/business/support/index? Page = content & id = TECH77071



12. Bug 11694127-rman duplicate not honoring TIME portion of date for "until time" [ID 11694127.8]

Symptoms:


During DUPLICATE, when the NLS_DATE_FORMAT does not contain the TIME specification,
The until time is truncated. Therefore, the constructed duplicate database may be at an incorrect time point.
For example:
Assume that RMAN replication uses the command: set until time "to_date ('Jan 27 2011 17:05:00 ', 'mon dd yyyy HH24: MI: ss ')";
The alert. log file will display: start until time 'Jan 27 2011 00:00:00 'using backup controlfile

Rediscovery Notes:
During the restoration, DUPLICATE fails because the NLS_DATE_FORMAT does not contain the TIME part and the until time is truncated.

Workaround
Set NLS_DATE_FORMAT to a format with time precision, and then
Run the RMAN Copy command using until time.
Example:
Setenv NLS_DATE_FORMAT 'dd-MON-YYYY HH24: MI: ss'
Setenv NLS_LANG 'American _ AMERICA. utf8'
Connect to the target and auxiliary (secondary) instances using RMAN
Execute the RMAN Copy command with UNTIL TIME

For affected/repaired versions, see Note 11694127.8