System slowdown caused by disk faults

OS: solaris 10

DBMS: Oracle 10.2.0.3.0
 
A carrier in Canada reported that the system was abnormal and slow, resulting in a backlog of data.
 
First generate an awr report

Log file sync 45,755 33,981 743 59.7 Commit
CPU time 14,009 24.6
Db file parallel write 63,119 11,374 180 20.0 System I/O
Db file sequential read 736,650 3,692 5 6.5 User I/O
Log file parallel write 9,148 3,081 337 5.4 System I/O
 

Most of them are caused by IO.
 
First Check log information
 
Select * from v $ logfile;
 
There are 64 50 m online log groups. Obviously unreasonable first
 
Add 5 2g log groups \
 
Alter databae add logfile group 66 '/filepath/redolog66.log' size 2g;
 
.
 
Alter database drop logfile group 1;
Alter database drop logfile group 2;
..
 
Alter database drop logfile group 64;
Alter database drop logfile group 65;
 
The problem persists.
 
Fortunately, we found an awr report from the previous month, which is far less load than the previous one.
 
Observe System IO
 
Device r/s w/s kr/s kw/s wait actv svc_t % w % B

Md0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md5 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md10 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md11 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md13 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md15 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md20 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md21 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md23 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Md25 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Sd0 0.0 0.2 0.0 0.1 0.0 0.0 3.9 0 0

Sd1 0.0 0.2 0.0 0.1 0.0 0.0 4.2 0 0

Sd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Sd3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Sd4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Ssd5 13.0 106.0 126.4 847.7 0.0 1.8 15.4 0 100

Ssd6 0.0 3.4 0.0 1.8 0.0 0.1 34.0 0 2

Ssd7 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Ssd8 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Ssd9 0.0 0.0 0.0 0.0 0.0 0.0 0 0

Nfs1 0.0 0.0 0.0 0.0 0.0 0.0 0 0
 

The write per second is only about KB, which is far from the expected performance of a magnetic array.
 
However, the performance and read performance of a large cp file are still acceptable. Inform frontline colleagues to check the device status first
 
I will introduce the cache feature of the magnetic array in other logs.
 
The feedback is that the cache battery has expired and the cache is disabled. Replace the battery.
 
End!