Oracle 11g cluster troubleshooting

Oracle 11g cluster troubleshooting

Symptom:
The Oracle cluster does not seem to be drifting IP addresses. The number of all session connections is only concentrated on one node. If this node is disconnected, it will not be automatically connected to another node;

Troubleshooting:

Run the cluster status check Command on node RAC1 (pay attention to the red font Section ):
Grid @ rac01: [/home/grid] crsctl stat res-t
--------------------------------------------------------------------------------
Name target state server STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
Ora. ARDATA. dg
ONLINE rac01
ONLINE rac02
Ora. CLDATA. dg
ONLINE rac01
ONLINE rac02
Ora. LISTENER. lsnr
ONLINE rac01
ONLINE rac02
Ora. USDATA. dg
ONLINE rac01
ONLINE rac02
Ora. USDATA01.dg
Online offline rac01
ONLINE rac02
Ora. asm
ONLINE rac01 Started
ONLINE rac02 Started
Ora. gsd
OFFLINE rac01
OFFLINE rac02
Ora. net1.network
ONLINE rac01
ONLINE rac02
Ora. ons
ONLINE rac01
ONLINE rac02
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
Ora. LISTENER_SCAN1.lsnr
1 ONLINE rac02
Ora. cvu
1 ONLINE rac02
Ora. productname
1 ONLINE rac02
Ora. rac01.vip
1 ONLINE rac01
Ora. rac02.vip
1 ONLINE rac02
Ora. racdb. db
1 online offline Instance Shutdown
2 ONLINE rac02 Open
Ora. scan1.vip
1 ONLINE rac02

 

As a result, the USDATA01 volume group may not work properly. Start the USDATA01 volume group and continue to report an error.

Grid @ rac01: [/home/grid] srvctl start diskgroup-g USDATA01
PRCR-1079: Failed to start resource ora. USDATA01.dg
CRS-5017: The resource action "ora. USDATA01.dg start" encountered the following error:
ORA-15032: not all alterations saved med
ORA-15017: diskgroup "USDATA01" cannot be mounted
ORA-15063: ASM discovered an insufficient number of disks for diskgroup "USDATA01"
. For details refer to "(: CLSN00107 :)" in "/u01/app/11.2.0/grid/log/rac01/agent/crsd/oraagent_grid/oraagent_grid.log ".
CRS-2674: Start of 'ora. USDATA01.dg 'on 'rac01' failed

 

View the ASM disk volume group and find that the USDATA01 volume group cannot be recognized normally. Therefore, the node RAC1 status is abnormal because it cannot identify the volume group properly.

Grid @ rac01: [/home/grid] asmcmd
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
Mounted extern n 512 4096 1048576 666603 0 332494 n ardata/
Mounted extern n 512 4096 1048576 51207 0 50811 y cldata/
Mounted extern n 512 4096 1048576 512009 0 124302 0 n usdata/