ASM Single Instance CRS-4124, CRS-4000 error handling

ASM Single Instance CRS-4124, CRS-4000 error handling

Install GI. Due to limited notebook resources, you have installed Oracle11g GI so that you can learn ASM. After the installation is complete, everything works normally.
However, the following error is reported after it is started today:
[Root @ myrac1 ~] # Su-grid
[Grid @ myrac1 ~] $ Crsctl start has

CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.
CRS-4124: Oracle High Availability Services startup failed.
CRS-4000: Command Start failed, or completed with errors.

View ohasd. log logs
18:02:42. 143: [UiServer] [2762939248] processMessage called
18:02:42. 144: [UiServer] [2762939248] Sending message to PE. ctx = 0xa8be6268
18:02:42. 144: [UiServer] [2762939248] Sending command to PE: 51
18:02:42. 144: [CRSPE] [2767141744] Processing PE command id = 158. Description: [Stat Resource: 0xb1df7aa0]
18:02:42. 392: [CRSPE] [2767141744] PE Command [Stat Resource: 0xb1df7aa0] has completed
18:02:42. 393: [CRSPE] [2767141744] UI Command [Stat Resource: 0xb1df7aa0] is replying to sender.
18:02:42. 395: [UiServer] [2762939248] Done for ctx = 0xa8be6268
18:02:42. 417: [UiServer] [2756705136] Closed: remote end failed/disc.
18:02:45. 055: [UiServer] [2756705136] S (0xa7fd3958): set Properties (grid, 0xb49c820)
18:02:45. 055: [UiServer] [2756705136] S (0xa8bae2d8): Accepted client connection: saddr = (ADDRESS = (PROTOCOL = ipc) (DEV = 36) (KEY = CRSD_UI_SOCKET )) daddr = (ADDRESS = (PROTOCOL = ipc) (KEY = CRSD_UI_SOCKET ))
18:02:45. 066: [UiServer] [2762939248] processMessage called
18:02:45. 066: [UiServer] [2762939248] Sending message to PE. ctx = 0xa8be8d90
18:02:45. 067: [UiServer] [2762939248] Sending command to PE: 52
18:02:45. 067: [CRSPE] [2767141744] Processing PE command id = 159. Description: [Stat Resource: 0xa45323b0]
18:02:45. 092: [CRSPE] [2767141744] PE Command [Stat Resource: 0xa45323b0] has completed
18:02:45. 093: [CRSPE] [2767141744] UI Command [Stat Resource: 0xa45323b0] is replying to sender.
18:02:45. 107: [UiServer] [2762939248] Done for ctx = 0xa8be8d90
18:02:45. 115: [UiServer] [2756705136] Closed: remote end failed/disc.
18:02:46. 416: [UiServer] [2756705136] S (0xa7fd3958): set Properties (grid, 0xb49c788)
18:02:46. 416: [UiServer] [2756705136] S (0xa8bae2d8): Accepted client connection: saddr = (ADDRESS = (PROTOCOL = ipc) (DEV = 36) (KEY = CRSD_UI_SOCKET )) daddr = (ADDRESS = (PROTOCOL = ipc) (KEY = CRSD_UI_SOCKET ))
18:02:46. 427: [UiServer] [2762939248] processMessage called
18:02:46. 428: [UiServer] [2762939248] Sending message to PE. ctx = 0xa8bb23b0
18:02:46. 428: [UiServer] [2762939248] Sending command to PE: 53
18:02:46. 428: [CRSPE] [2767141744] Processing PE command id = 160. Description: [Stat Resource: 0xa1_f3b8]
18:02:46. 436: [CRSPE] [2767141744] PE Command [Stat Resource: 0xa0000f3b8] has completed
18:02:46. 437: [CRSPE] [2767141744] UI Command [Stat Resource: 0xa0000f3b8] is replying to sender.
18:02:46. 438: [UiServer] [2762939248] Done for ctx = 0xa8bb23b0
18:02:46. 460: [UiServer] [2756705136] Closed: remote end failed/disc.
View startup status of related services
[Grid @ myrac1 ohasd] $ ps-ef | grep cssd
Grid 5402 4816 0 00:00:00 pts/3 grep cssd
[Grid @ myrac1 ohasd] $ ps-ef | grep has
Grid 2857 1 1 17: 29? 00:00:34/g01/app/grid/product/11.2.0/grid/bin/ohasd. bin reboot
Grid 5432 4816 0 00:00:00 pts/3 grep has
[Grid @ myrac1 ohasd] $ ps-ef | grep d. bin
Grid 2857 1 1 17: 29? 00:00:34/g01/app/grid/product/11.2.0/grid/bin/ohasd. bin reboot
Grid 3028 1 0 :31? 00:00:01/g01/app/grid/product/11.2.0/grid/bin/tnslsnr LISTENER-inherit
Grid 3140 1 0 :32? 00:00:20/g01/app/grid/product/11.2.0/grid/bin/oraagent. bin
Grid 3206 1 0 :32? 00:00:04/g01/app/grid/product/11.2.0/grid/bin/cssdagent
Grid 3240 1 0 :32? 00:00:03/g01/app/grid/product/11.2.0/grid/bin/orarootagent. bin
Grid 3253 1 0 :32? 00:00:14/g01/app/grid/product/11.2.0/grid/bin/diskmon. bin-d-f
Grid 5461 4816 1 00:00:00 pts/3 grep d. bin

It is found that the has service is not started. It is automatically started when it is started. The init. ohasd run command should be automatically executed.
[Grid @ myrac1 ohasd] $ cat/etc/inittab | grep ohasd
H1: 35: respawn:/etc/init. d/init. ohasd run>/dev/null 2> & 1
O official explanation:
With Oracle Clusterware 11g Release 2 (11.2 ), cluster commands have been introduced that allow stopping the cluster stack on a remote note (as opposed to stopping it locally only with the commands listed above ). in order to stop the cluster stack with the exception of the Oracle High Availability Services (OHAS, daemon OHASD), use: crsctl stop cluster-all # stops the cluster layer on all servers in the cluster crsctl stop cluster-n # stops the cluster layer the named server
At 11g, ohasd includes crsd, ocssd, and evmd.11g cluster in two layers: lower stack and higher stack.
Ohasd is responsible for starting the lower stack cluster resources, and crsd is responsible for starting the upper-layer cluster resources.

Since the ohasd service is not started, it is started manually.
[Root @ myrac1 ~] #/Etc/init. d/init. ohasd run
Mkfifo: cannot create fifo '/var/tmp/. oracle/npohasd': File exists
It has been running and has not been terminated. It is similar to the running method of tomcat, but it can be run in the background and added.
Check the resource startup status later
[Grid @ myrac1 ohasd] $ crsctl status res-t
--------------------------------------------------------------------------------
Name target state server STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
Ora. DATA_DG.dg ONLINE myrac1
Ora. DG_FRA.dg ONLINE myrac1
Ora. LISTENER. lsnr ONLINE myrac1
Ora. SYS_DG.dg ONLINE myrac1
Ora. asm ONLINE myrac1 Started
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
Ora.css d ONLINE myrac1
Ora. diskmon ONLINE myrac1
Ora. hjj. db OFFLINE Instance Shutdown
[Grid @ myrac1 ohasd] $ crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora. DATA_DG.dg ora... up. type ONLINE myrac1
Ora. DG_FRA.dg ora... up. type ONLINE myrac1
Ora... ER. lsnr ora... er. type ONLINE myrac1
Ora. SYS_DG.dg ora... up. type ONLINE myrac1
Ora. asm ora. asm. type ONLINE myrac1
Ora.css d. type ONLINE myrac1
Ora. diskmon ora... on. type ONLINE myrac1
Ora. hjj. db ora... se. type OFFLINE
Only ora. hjj. db is offline, because the database has not been started, it will become ONLINE after it is started.
Start the ASM instance
[Grid @ myrac1 ~] $ Sqlplus/as sysasm

SQL * Plus: Release 11.2.0.1.0 Production on Wed Feb 19 17:34:03 2014

Copyright (c) 1982,200 9, Oracle. All rights reserved.

Connected:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0-Production
With the Automatic Storage Management option

SQL> startup
ORA-01081: cannot start already-running ORACLE-shut it down first
SQL> shutdown immediater
SP2-0717: illegal SHUTDOWN option
SQL> shutdown immediate
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup
ASM instance started

Total System Global Area 284565504 bytes
Fixed Size 1336036 bytes
Variable Size 258063644 bytes
ASM Cache 25165824 bytes
ASM diskgroups mounted

Start Database
[Oracle @ myrac1 ~] $ Sqlplus/as sysdba

SQL * Plus: Release 11.2.0.1.0 Production on Wed Feb 19 18:40:55 2014

Copyright (c) 1982,200 9, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup
ORACLE instance started.

Total System Global Area 313860096 bytes
Fixed Size 1336232 bytes
Variable Size 130026584 bytes
Database Buffers 176160768 bytes
Redo Buffers 6336512 bytes
Database mounted.
Database opened.
View resource startup status
[Grid @ myrac1 ~] $ Crs_stat-t
Name Type Target State Host
------------------------------------------------------------
Ora. DATA_DG.dg ora... up. type ONLINE myrac1
Ora. DG_FRA.dg ora... up. type ONLINE myrac1
Ora... ER. lsnr ora... er. type ONLINE myrac1
Ora. SYS_DG.dg ora... up. type ONLINE myrac1
Ora. asm ora. asm. type ONLINE myrac1
Ora.css d. type ONLINE myrac1
Ora. diskmon ora... on. type ONLINE myrac1
Ora. hjj. db ora... se. type ONLINE myrac1

If a resource is not started, use crsctl start res resource_name to start it.
The problem has been solved.

Conclusion: when starting the service, you must always pay attention to the background logs and actions to clearly identify the link in which errors occur, so as to quickly determine the problem and solve the problem.

Appendix: Common commands
Check the has startup status
Crsctl check has
Check the startup status of css
Crsctl check css
Check resource startup status
Crs_stat-t-v
Crsctl status res-t
Start a resource
Crsctl start res resource_name
Ocr Information
Ocrcheck
View the hjj configuration information of the database
Srvctl config database-d hjj
View the parameters of a resource
Crs_stat-p ora. hjj. db
Crsctl command usage
[Grid @ myrac1 ~] $ Crsctl-h
Usage: crsctl add-add a resource, type or other entity
Crsctl check-check a service, resource or other entity
Crsctl config-output autostart configuration
Crsctl debug-obtain or modify debug state
Crsctl delete-delete a resource, type or other entity
Crsctl disable-disable autostart
Crsctl enable-enable autostart
Crsctl get-get an entity value
Crsctl getperm-get entity permissions
Crsctl lsmodules-list debug modules
Crsctl modify-modify a resource, type or other entity
Crsctl query-query service state
Crsctl pin-Pin the nodes in the nodelist
Crsctl relocate-relocate a resource, server or other entity
Crsctl replace-replaces the location of voting files
Crsctl setperm-set entity permissions
Crsctl set-set an entity value
Crsctl start-start a resource, server or other entity
Crsctl status-get status of a resource or other entity
Crsctl stop-stop a resource, server or other entity
Crsctl unpin-unpin the nodes in the nodelist
Crsctl unset-unset a entity value, restoring its default

Oracle High Availability service version released
Crsctl query has releaseversion
Oracle High Availability service version
Crsctl query has softwareversion