Oracle 11g RAC CRS-4535/ORA-15077

After Oracle 11g RAC is installed, why does crsd on the second node fail to start? The error message is CRS-4535: cannot communicate with cluster ready services. The specific error information must be viewed in the crsd. log.

1. Environment [root @ linux2 ~] # Cat/etc/issue Enterprise Linux Server Release 5.5 (Carthage) kernel \ r on an \ m [root @ linux2 bin] #. /crsctl query CRS activeversion Oracle clusterware active version on the cluster is [11.2.0.1.0] # note that grid and root users are used to operate on different objects in the following description. 2. Error symptoms [root @ linux2 bin] #. /crsctl check CRS CRS-4638: Oracle High Availability services is online CRS-4535: cannot communicate with cluster ready services # CRS-4535 CRS-4529: Cluster synchronization services is online CRS-4533: event manager is online [root @ linux2 bin] # ps-Ef | grep D. bin # The following query does not contain crsd. bin root 3886 1 1 09:50? 00:00:11/u01/APP/11.2.0/GRID/bin/ohasd. Bin reboot grid 3938 1 0 09:51? 00:00:04/u01/APP/11.2.0/GRID/bin/oraagent. Bin grid 4009 1 0? 00:00:00/u01/APP/11.2.0/GRID/bin/gipcd. Bin grid 4014 1 0? 00:00:00/u01/APP/11.2.0/GRID/bin/mdnsd. Bin grid 4028 1 0? 00:00:02/u01/APP/11.2.0/GRID/bin/gpnpd. Bin root 4040 1 0? 00:00:03/u01/APP/11.2.0/GRID/bin/cssdmonitor root 4058 1 0? 00:00:04/u01/APP/11.2.0/GRID/bin/cssdagent root 4060 1 0? 00:00:00/u01/APP/11.2.0/GRID/bin/orarootagent. Bin grid 4090 1 2 09:51? 00:00:15/u01/APP/11.2.0/GRID/bin/ocssd. Bin grid 4094 1 0? 00:00:02/u01/APP/11.2.0/GRID/bin/diskmon. Bin-D-F root 4928 1 0? 00:00:00/u01/APP/11.2.0/GRID/bin/octssd. Bin reboot grid 4945 1 0 09:51? 00:00:02/u01/APP/11.2.0/GRID/bin/evmd. bin root 6514 5886 0 00:00:00 pts/1 grep D. bin [root @ linux2 bin] #. /crsctl stat res-T-init ready name target State server state_details analyze cluster resources ------------------------------------------- ------------------------------------- Ora. ASM 1 online linux2 cluster reconfigura tion ora. crsd 1 online offline # crsd in Offline state ora.css D 1 online linux2 ora.css dmonitor 1 online linux2 ora. ctssd 1 online linux2 observer ora. diskmon 1 online linux2 ora. drivers. ACFs 1 online offline # ACFs is in the offline status ora. evmd 1 online linux2 ora. gipcd 1 online linux2 Ora. gpnpd 1 online linux2 ora. mdnsd 1 online linux2 # view the log file corresponding to crsd [grid @ linux2 ~] $ View $ ORACLE_HOME/log/linux2/crsd. log 10:28:27. 107: [gipcxcpt] [1768145488] gipcshutdownf: Skipping shutdown, count 1, from [clsgpn1_c: 1021], RET gipcretsuccess (0) 10:28:27. 107: [ocrasm] [1768145488] proprasmo: Error in open/create file in DG [ocr_vote] # opening a disk group error [ocrasm] [1768145488] SLOS: cat = 7, OPN = kgfoal06, DEP = 15077, Loc = kgfokge ORA-15077: cocould not locate ASM instance serving a required diskgroup # ora error 10:28:27. 107: [ocrasm] [1768145488] proprasmo: kgfocheckmount returned [7] 10:28:27. 107: [ocrasm] [1768145488] proprasmo: the ASM instance is down # The instance is closed 10:28:27. 107: [ocrraw] [1768145488] proprioo: failed to open [+ ocr_vote]. returned proprasmo () with [26]. marking location as unavailable. 10:28:27. 10 7: [ocrraw] [1768145488] proprioo: No OCR/OLR devices are usable # OCR/OLR device unavailable 10:28:27. 107: [ocrasm] [1768145488] proprasmcl: asmhandle is null 10:28:27. 107: [ocrraw] [1768145488] proprinit: cocould not open raw device 10:28:27. 107: [ocrasm] [1768145488] proprasmcl: asmhandle is null 10:28:27. 107: [ocrapi] [1768145488] a_init: 16!: Backend init unsuccessful: [26] 10:28:27. 107: [crsocr] [1768145488] OCR context init failure. error: PROC-26: Error while accessing the physical storage ASM error [SLOS: cat = 7, fig = kgfoal06, DEP = 15077, Loc = kgfokge ORA-15077: cocould not locate ASM instance serving a required diskgroup] [7] 10:28:27. 107: [crsd] [1768145488] [panic] crsd exiting: cocould not init OCR, code: 26 20 13-01-05 10:28:27. 107: [crsd] [1768145488] Done. [root @ linux2 bin] # ps-Ef | grep pmon # view the pmon process, it also indicates that the ASM instance does not start root 7447 7184 0 00:00:00 pts/2 grep pmon # As shown in the above analysis, it should be because the crsd process cannot be started because the ASM instance is not started. 3. Solve [grid @ linux2 ~] $ Asmcmd connected to an idle instance. asmcmd> startup # Start ASM instance started total system global area 283930624 bytes fixed size 2212656 bytes variable size 256552144 bytes ASM cache 25165824 bytes ASM diskgroups mounted asmcmd> exit # Author: robinson # blog: http://blog.csdn.net/robinson_0612 # view the status of cluster resources again [root @ linux2 bin] #. /crsctl stat res-T-init -------------------------------------------------------------------------------- name target State server state_details implements cluster resources restart ora. ASM 1 online linux2 started ora. crsd 1 online intermediate linux2 ora.css D 1 online linux2 ora.css dmonitor 1 online linux2 ora. ctssd 1 online linux2 observer ora. diskmon 1 online linux2 ora. drivers. ACFs 1 online offline ora. evmd 1 online linux2 ora. gipcd 1 online linux2 ora. gpnpd 1 online linux2 ora. mdnsd 1 online linux2 # Start ACFs [root @ linux2 bin] #. /crsctl start res ora. drivers. ACFs-init CRS-2672: attempting to start' ora. drivers. ACFs 'on 'linux2' CRS-2676: Start of 'ora. drivers. ACFs on 'linux2 'succeeded # Then all statuses are online [root @ linux2 bin] #. /crsctl stat res-T-init -------------------------------------------------------------------------------- name target State server state_details implements cluster resources restart ora. ASM 1 online linux2 started ora. crsd 1 online linux2 ora.css D 1 online linux2 ora.css dmonitor 1 online linux2 ora. ctssd 1 online linux2 observer ora. diskmon 1 online linux2 ora. drivers. ACFs 1 online linux2 ora. evmd 1 online linux2 ora. gipcd 1 online linux2 ora. gpnpd 1 online linux2 ora. mdnsd 1 online linux2

Grid-related fault links:
Troubleshooting crsd start up issue [ID 1323698.1]
How to troubleshoot grid infrastructure startup issues [ID 1050908.1]

More references

For more information about Oracle RAC, see
Use crs_setperm to modify the resource owner and permissions of RAC.
Use crs_profile to manage RAC resource configuration files
RAC database startup and Shutdown
Oracle RAC services
Services in Oracle Database 10g
Migrate datbase from single instance to Oracle RAC
Connect Oracle RAC to a specified instance
Oracle RAC load balancing test (combined with server and client)
Oracle RAC server connection Load Balance)
Load Balance)
Non-Default port listening configuration in Oracle RAC (listener. ora tnsnames. ora)
Oracle RAC Listener Configuration (listener. ora tnsnames. ora)
Configure RAC load balancing and Failover
CRS-1006, CRS-0215 fault case
Installing Oracle 10g RAC Based on Linux (RHEL 5.5)
Use runcluvfy to verify the Oracle RAC installation environment

For more information about the basics and concepts of Oracle network configuration, see:
Configure dynamic service registration for non-default ports
Configure sqlnet. ora to restrict IP Access to Oracle
Configure and manage Oracle listener logs
Set the Oracle listener password (listener)
Configure the Oracle client to connect to the database

For more information about user-managed backup and recovery, see
Oracle cold backup
Oracle Hot Backup
Concept of Oracle backup recovery
Oracle instance recovery
Oracle recovery based on user management
System tablespace management and Backup Recovery
Sysaux tablespace management and recovery
Oracle backup control file recovery (unsing backup controlfile)

For information on RMAN backup recovery and management, see
RMAN overview and architecture
RMAN configuration, Monitoring and Management
Detailed description of RMAN backup
RMAN restoration and recovery
Create and use RMAN catalog
Create RMAN storage script based on catalog
Catalog-based RMAN backup and recovery
RMAN backup path confusion
Use RMAN for recovery from different machine backups (WIN platform)
Use RMAN to migrate a file system database to ASM
Linux RMAN backup shell script
Use RMAN to migrate the database to a different machine

For the Oracle architecture, see
Oracle tablespace and data files
Oracle Password File
Oracle parameter file
Oracle online redo log file)
Oracle Control File)
Oracle archiving logs
Oracle rollback and undo)
Oracle database instance startup and Shutdown Process
Automated Management of Oracle 10g SGA
Oracle instances and Oracle databases (Oracle Architecture)