ORA-19502/ORA-27063 error in rman backup

RMAN> 2> 3> 4> 5> 6> 7> 8> 9> 10> 11> 12> 13> 14> 15> 16> 17> 18> 19> 20>

Allocated channel: t11

Channel t11: sid = 824 instance = ncdb1 devtype = DISK

 

Allocated channel: t12

Channel t12: sid = 838 instance = ncdb1 devtype = DISK

 

Allocated channel: t13

Channel t13: sid = 809 instance = ncdb1 devtype = DISK

 

Allocated channel: t14

Channel t14: sid = 886 instance = ncdb1 devtype = DISK

 

Allocated channel: t15

Channel t15: sid = 620 instance = ncdb1 devtype = DISK

 

Allocated channel: t16

Channel t16: sid = 599 instance = ncdb1 devtype = DISK

 

Allocated channel: t17

Channel t17: sid = 482 instance = ncdb1 devtype = DISK

 

Allocated channel: t18

Channel t18: sid = 506 instance = ncdb1 devtype = DISK

A total of 8 channels are activated

 

Channel t12: starting full datafile backupset

Channel t12: specifying datafile (s) in backupset

Input datafile fno = 00008 name =/dev/rnc32g_39

Input datafile fno = 00016 name =/dev/rnc32g_47

Input datafile fno = 00024 name =/dev/rnc32g_57

Input datafile fno = 00032 name =/dev/rnc32g_25

Input datafile fno = 00040 name =/dev/rnc32g_33

Input datafile fno = 00048 name =/dev/rnc32g_3

Input datafile fno = 00056 name =/dev/rnc32g_11

Input datafile fno = 00064 name =/dev/rnc32g_19

Input datafile fno = 00072 name =/dev/rnc32g_67

Input datafile fno = 00080 name =/dev/rnc32g_106

Input datafile fno = 00088 name =/dev/rnc32g_114

Input datafile fno = 00096 name =/dev/rnc32g_87

Input datafile fno = 00104 name =/dev/rnc32g_95

Input datafile fno = 00112 name =/dev/rnc32g_103

Input datafile fno = 00120 name =/dev/rnc32g_75

Input datafile fno = 00003 name =/dev/rnc50_sysaux

Input datafile fno = 00130 name =/dev/rnc32g_119

Channel t12: starting piece 1 at 14-MAY-12

-- Channel 12: Back up data files

 

Channel t17: starting full datafile backupset

Channel t17: specifying datafile (s) in backupset

Input datafile fno = 00002 name =/dev/rnc32g_22

Input datafile fno = 00013 name =/dev/rnc32g_44

Input datafile fno = 00021 name =/dev/rnc32g_54

Input datafile fno = 00029 name =/dev/rnc32g_62

Input datafile fno = 00037 name =/dev/rnc32g_30

Input datafile fno = 00045 name =/dev/rnc32g_38

Input datafile fno = 00053 name =/dev/rnc32g_8

Input datafile fno = 00061 name =/dev/rnc32g_16

Input datafile fno = 00069 name =/dev/rnc32g_64

Input datafile fno = 00077 name =/dev/rncundo_33g_4

Input datafile fno = 00085 name =/dev/rnc32g_111

Input datafile fno = 00093 name =/dev/rnc32g_84

Input datafile fno = 00101 name =/dev/rnc32g_92

Input datafile fno = 00109 name =/dev/rnc32g_100

Input datafile fno = 00117 name =/dev/rnc32g_72

Input datafile fno = 00006 name =/dev/rnc50_4g_1

Channel t17: starting piece 1 at 14-MAY-12

-- Channel 17: Back up data files

 

Channel t15: finished piece 1 at 15-MAY-12

Piece handle =/rman/db_mpnb04jl_1_1 tag = TAG20120514T204954 comment = NONE

Channel t15: backup set complete, elapsed time: 06:07:59

Channel t11: finished piece 1 at 15-MAY-12

Piece handle =/rman/db_mlnb04jk_1_1 tag = TAG20120514T204954 comment = NONE

Channel t11: backup set complete, elapsed time: 06:17:25

Channel t16: finished piece 1 at 15-MAY-12

Piece handle =/rman/db_mqnb04jm_1_1 tag = TAG20120514T204954 comment = NONE

Channel t16: backup set complete, elapsed time: 06:34:49

Channel t14: finished piece 1 at 15-MAY-12

Piece handle =/rman/db_monb04jl_1_1 tag = TAG20120514T204954 comment = NONE

Channel t14: backup set complete, elapsed time: 06:40:05

Channel t18: finished piece 1 at 15-MAY-12

Piece handle =/rman/db_msnb04jn_1_1 tag = TAG20120514T204954 comment = NONE

Channel t18: backup set complete, elapsed time: 06:43:38

Channel t13: finished piece 1 at 15-MAY-12

Piece handle =/rman/db_mnnb04jl_1_1 tag = TAG20120514T204954 comment = NONE

Channel t13: backup set complete, elapsed time: 07:40:56

-- We can see that the rman backup has completed the Channel 11/13/14/15/16/18, that is, channel 12/17 has not been completed so far.

 

RMAN-03009: failure of backup command on t12 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30481025 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Additional information:-1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30480897 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Channel t12 disabled, job failed on it will be run on another channel

-- Channel 12 error (insufficient disk space)

 

Channel t11: starting full datafile backupset

Channel t11: specifying datafile (s) in backupset

Input datafile fno = 00008 name =/dev/rnc32g_39

Input datafile fno = 00016 name =/dev/rnc32g_47

Input datafile fno = 00024 name =/dev/rnc32g_57

Input datafile fno = 00032 name =/dev/rnc32g_25

Input datafile fno = 00040 name =/dev/rnc32g_33

Input datafile fno = 00048 name =/dev/rnc32g_3

Input datafile fno = 00056 name =/dev/rnc32g_11

Input datafile fno = 00064 name =/dev/rnc32g_19

Input datafile fno = 00072 name =/dev/rnc32g_67

Input datafile fno = 00080 name =/dev/rnc32g_106

Input datafile fno = 00088 name =/dev/rnc32g_114

Input datafile fno = 00096 name =/dev/rnc32g_87

Input datafile fno = 00104 name =/dev/rnc32g_95

Input datafile fno = 00112 name =/dev/rnc32g_103

Input datafile fno = 00120 name =/dev/rnc32g_75

Input datafile fno = 00003 name =/dev/rnc50_sysaux

Input datafile fno = 00130 name =/dev/rnc32g_119

Channel t11: starting piece 1 at 15-MAY-12

-- After channel 12 reports an error, Channel 11 has completed the last backup, so the data file with the error of backup channel 12 is started.

 

RMAN-03009: failure of backup command on t17 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753793 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Additional information:-1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753665 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Channel t17 disabled, job failed on it will be run on another channel

-- Channel 17 also reports an error because of disk space

 

Channel t13: starting full datafile backupset

Channel t13: specifying datafile (s) in backupset

Input datafile fno = 00002 name =/dev/rnc32g_22

Input datafile fno = 00013 name =/dev/rnc32g_44

Input datafile fno = 00021 name =/dev/rnc32g_54

Input datafile fno = 00029 name =/dev/rnc32g_62

Input datafile fno = 00037 name =/dev/rnc32g_30

Input datafile fno = 00045 name =/dev/rnc32g_38

Input datafile fno = 00053 name =/dev/rnc32g_8

Input datafile fno = 00061 name =/dev/rnc32g_16

Input datafile fno = 00069 name =/dev/rnc32g_64

Input datafile fno = 00077 name =/dev/rncundo_33g_4

Input datafile fno = 00085 name =/dev/rnc32g_111

Input datafile fno = 00093 name =/dev/rnc32g_84

Input datafile fno = 00101 name =/dev/rnc32g_92

Input datafile fno = 00109 name =/dev/rnc32g_100

Input datafile fno = 00117 name =/dev/rnc32g_72

Input datafile fno = 00006 name =/dev/rnc50_4g_1

Channel t13: starting piece 1 at 15-MAY-12

-- Channel 13 also tries to back up data files that failed to be created through channel 17

 

RMAN-03009: failure of backup command on t11 channel at 05/15/2012 04:39:59

ORA-19504: failed to create file "/rman/db_mtnb1_u_1_1"

ORA-27044: unable to write the header block of file

Ibm aix risc System/6000 Error: 28: No space left on device

Additional information: 3

Addition

-- Because there is no free space, Channel 11 is terminated,

-- At this time, rman terminates abnormally, resulting in the subsequent termination records of Channel 13 not being printed to the log.

After reading the rman log, it is easy to understand that a series of errors are caused by insufficient disk space to store the rman backup.
 
Check the remaining disk space
 

 

 

Filesystem GB blocks Free % Used Iused % Iused Mounted on

/Dev/hd4 10.00 9.75 3% 6548 1%/

/Dev/hd2 10.00 4.55 55% 84383 8%/usr

/Dev/hd9var 5.00 4.04 20% 6290 1%/var

/Dev/hd3 5.00 3.87 23% 1551 1%/tmp

/Dev/hd1 10.00 9.91 1% 382 1%/home

/Proc---/proc

/Dev/hd10opt 5.00 4.89 3% 3502 1%/opt

/Dev/archalv 99.00 82.98 17% 96 1%/archa

/Dev/fslv01 40.00 19.49 52% 72324 2%/ora10

/Dev/fslv00 1800.00 467.25 75% 10 1%/rman

This is confusing, and there are still 467.25 GB of disk space. How can I report an error?
 
Cause Analysis
 

 

 

RMAN-03009: failure of backup command on t12 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30481025 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Additional information:-1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mmnb04jl_1_1", blockno 30480897 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Channel t12 disabled, job failed on it will be run on another channel

 

RMAN-03009: failure of backup command on t17 channel at 05/15/2012 04:39:58

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753793 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Additional information:-1

Additional information: 1048576

ORA-19502: write error on file "/rman/db_mrnb04jm_1_1", blockno 30753665 (blocksize = 8192)

ORA-27063: number of bytes read/written is incorrect

Ibm aix risc System/6000 Error: 28: No space left on device

Channel t17 disabled, job failed on it will be run on another channel

When the two channels are written into the rman backup to the disk, they found that the disk space was insufficient at 04:39:58 on, and the two channels were prepared to write block 30480897/30753665 respectively, the number of data blocks written by the two channels is 30480896/30753664, and the size of the written files is (30480896 + 30753664) * 8192/1024/1024/1024 = 467.1826171875G. here we can see that the remaining disk space is 467.25 GB, in fact, 467.1826171875 GB has been written at that time, and an error occurred while continuing to write. then, to ensure the correctness of the backup, rman automatically deletes the backup file with the 467.1826171875G error that has been backed up at that time. therefore, after the backup is complete, we can see that there is still a large amount of disk space remaining and the rman package space is insufficient.