ORA-3136 error and AIX system 3D32B80D error caused by high system load during Oracle backup

This is because the database is backed up by rman at every day, resulting in frequent I/O reads of the database, resulting in a large amount of CPU waiting. IO, excessive CPU wait, resulting in load comparison of the entire server

This is because the database is backed up by rman at every day, resulting in frequent I/O reads of the database, resulting in a large amount of CPU waiting. IO, excessive CPU wait, resulting in load comparison of the entire server

Environment:

Database: 64-bit Oracle 10.2.0.1.0 under AIX5.3

TSM5.4: Windows 2003

With database: IBM3100

1. Database alert_SID.log

Mon Jan 11 20:05:38 2010

Control autobackup written to SBT_TAPE device

Comment 'api Version 2.0, MMS Version5.4.1.0 ',

Media '123'

Handle 'C-3524587154-20100111-05'

Mon Jan 11 22:12:32 2010

WARNING: inbound connection timed out (ORA-3136)

Mon Jan 11 22:12:33 2010

WARNING: inbound connection timed out (ORA-3136)

2. Errors in the AIX system:

> Errpt | more

Identifier timestamp t c RESOURCE_NAME DESCRIPTION

3D32B80D 0121222410 p s topsvcs NIM thread blocked

3D32B80D 0121222410 p s topsvcs NIM thread blocked

View Details:

> Errpt-aj 3D32B80D | more

3D32B80D error is mainly caused by the high load of the entire system, resulting in HA heartbeat failure to communicate normally.

Specific reasons:

This is because the database is backed up by rman at every day, resulting in frequent I/O reads of the database, resulting in a large amount of CPU waiting. IO and excessive CPU waiting results in a high load on the entire server. The database server has a high load during the backup time period. As a result, the database user cannot connect to the database within 60 seconds, and the connection timeout (timeout) occurs ).

As a result, the ORA-3136 error in the above aler_SID.log appears.

A problem occurs when Oracle 10 Gb database is backed up to the database with RMAN. You can use the metalink account of Oracle to view Note 311068.1

Test the database, back up data from disks to disks, back up data from disks to libraries, and back up data from read-only disks according to Note 311068.1. At last, we found that the amount of data in the data files read from the system during Backup is too high (some disks are busy with IO up to 100%). We can solve this problem by using rman to read the disk at the current speed during Backup.

The script is adjusted as follows:

Run {

Allocate channel t1 type 'sbt _ tape 'parms' ENV = (TDPO_OPTFILE =/usr/tivoli/tsm/client/oracle/bin64/tdpo. opt) 'rate20m;

Allocate channel t2 type 'sbt _ tape 'parms' ENV = (TDPO_OPTFILE =/usr/tivoli/tsm/client/oracle/bin64/tdpo. opt) 'rate20m;

Backup incremental level 1 cumulative filesperset 10

FORMAT 'incr _ % T _ % s _ % P' DATABASE;

SQL 'alter system archive log current ';

Backup filesperset 10 MAXSETSIZE10G

FORMAT 'arch _ % T _ % s _ % P'

SKIP INACCESSIBLE

Archivelog all delete input;

Release channel t1;

Release channel t2;

}