Solution for CRS status unknown in rac10g

Crs_stop-all

Crs_stop-all Crs_stop-all When crs_stop-all is executed in RAC, the system prompts that some services cannot be stopped. When crs_stat-T is used to check the status, the state of the services that cannot be stopped is unknown, The service cannot start with crs_start in the unknown state. For example: [Oracle @ Rac1 ~] $ Crs_stop-all Attempting to stop 'ora. rac1.gsd' on member 'rac1' Attempting to stop 'ora. rac1.ons 'on member 'rac1' Attempting to stop 'ora. rac2.gsd' on member 'rac2' Attempting to stop 'ora. ora9i. db' on member 'rac1' Attempting to stop 'ora. rac2.ons 'on member 'rac2' Stop of 'ora. rac1.gsd' on member 'rac1' succeeded. Stop of 'ora. rac2.gsd' on member 'rac2' succeeded. Stop of 'ora. rac2.ons 'on member 'rac2' succeeded. Stop of 'ora. rac1.ons 'on member 'rac1' succeeded. Stop of 'ora. ora9i. DB 'on member 'rac1' succeeded. Attempting to stop 'ora. rac1.listener _ rac1.lsnr 'on member 'rac1' Attempting to stop 'ora. rac2.listener _ rac2.lsnr 'on member 'rac2' 'Ora. rac1.listener _ rac1.lsnr 'on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. 'Ora. ora9i. ora9i1. Inst 'is already offline. 'Ora. rac2.listener _ rac2.lsnr 'on member 'rac2' has experienced an unrecoverable failure. Human intervention required to resume its availability. 'Ora. ora9i. ora9i2. Inst 'is already offline. Attempting to stop 'ora. rac1.asm1. ASM 'on member 'rac1' 'Ora. rac1.asm1. ASM 'on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop 'ora. rac2.asm2. ASM 'on member 'rac2' Attempting to stop 'ora. rac1.listener _ rac1.lsnr 'on member 'rac1' 'Ora. rac2.asm2. ASM 'on member 'rac2' has experienced an unrecoverable failure. Human intervention required to resume its availability. 'Ora. rac1.listener _ rac1.lsnr 'on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop 'ora. rac1.asm1. ASM 'on member 'rac1' 'Ora. rac1.asm1. ASM 'on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop 'ora. rac2.listener _ rac2.lsnr 'on member 'rac2' Attempting to stop 'ora. rac1.vip 'on member 'rac1' 'Ora. rac2.listener _ rac2.lsnr 'on member 'rac2' has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop 'ora. rac2.asm2. ASM 'on member 'rac2' 'Ora. rac2.asm2. ASM 'on member 'rac2' has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to stop'Ora. rac2.vip'On member 'rac2' Stop of 'ora. rac1.vip 'on member 'rac1' succeeded. Stop'Ora. rac2.vip'On member 'rac2' succeeded. CRS-0216: Cocould not stop resource 'ora. ora9i. ora9i1. inst '. CRS-0216: Cocould not stop resource 'ora. ora9i. ora9i2. inst '. CRS-0216: Cocould not stop resource 'ora. rac1.asm1. ASM '. CRS-0216: Cocould not stop resource 'ora. rac1.listener _ rac1.lsnr '. CRS-0216: Cocould not stop resource 'ora. rac1.vip '. CRS-0216: Cocould not stop resource 'ora. rac2.asm2. ASM '. CRS-0216: Cocould not stop resource 'ora. rac2.listener _ rac2.lsnr '. CRS-0216: Cocould not stop resource'Ora. rac2.vip'.   In the unknown state, the crs_start-all command will not start RAC, and an error will be reported during startup. [Oracle @ Rac1 ~] $ Crs_start-all Attempting to start 'ora. rac1.vip 'on member 'rac1' Attempting to start'Ora. rac2.vip'On member 'rac2' Attempting to start 'ora. rac2.gsd' on member 'rac2' Attempting to start 'ora. rac2.ons 'on member 'rac2' Attempting to start 'ora. rac1.gsd' on member 'rac1' Attempting to start 'ora. rac1.ons 'on member 'rac1' 'Ora. rac2.gsd' on member 'rac2' has experienced an unrecoverable failure. Human intervention required to resume its availability. 'Ora. rac2.ons 'on member 'rac2' has experienced an unrecoverable failure. Human intervention required to resume its availability. Attempting to start 'ora. ora9i. db' on member 'rac1' 'Ora. rac1.ons 'on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. 'Ora. rac1.gsd' on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. 'Ora. ora9i. db' on member 'rac1' has experienced an unrecoverable failure. Human intervention required to resume its availability. Start of 'ora. rac1.vip 'on member 'rac1' succeeded. CRS-1028: dependency analysis failed because: 'Resource in unknown state: ora. rac1.asm1. ASM' Start'Ora. rac2.vip'On member 'rac2' succeeded. CRS-1028: dependency analysis failed because: 'Resource in unknown state: ora. rac2.asm2. ASM' CRS-0215: cocould not start resource 'ora. ora9i. db '. CRS-0223: Resource 'ora. ora9i. ora9i1. Inst 'has placement error. CRS-0223: Resource 'ora. ora9i. ora9i2. Inst 'has placement error. CRS-0215: cocould not start resource 'ora. rac1.gsd '. CRS-0215: cocould not start resource 'ora. rac1.ons '. CRS-0215: cocould not start resource 'ora. rac2.gsd '. CRS-0215: cocould not start resource 'ora. rac2.ons '. At this time, crs_stat-t still has many unknown states. (Annoying, annoying, and annoying. It's really strange in RAC. when the service is shut down normally, The status can only be online or offline (started or not started). In the result, an unknown is displayed in RAC ): [Oracle @ Rac1 ~] $ Crs_stat-T Name type target State host ------------------------------------------------------------ Ora. ora9i. DB application online unknown Rac1 Ora... i1.inst application offline Ora... i2.inst application offline Ora... sm1.asm application offline unknown Rac1 Ora... c1.lsnr application offline unknown Rac1 Ora. rac1.gsd application online unknown Rac1 Ora. rac1.ons application online unknown Rac1 Ora. rac1.vip application online Rac1 Ora... sm2.asm application offline unknown rac2 Ora... c2.lsnr application offline unknown rac2 Ora. rac2.gsd application online unknown rac2 Ora. rac2.ons application online unknown rac2 Ora. rac2.vipApplication online rac2 [Oracle @ Rac1 ~] $ Sqlplus/nolog SQL * Plus: Release 10.2.0.1.0-production on Thu Aug 7 13:42:36 2008 Copyright (c) 1982,200 5, Oracle. All rights reserved. SQL> Conn/As sysdba Connected to an idle instance. It indicates that the database is no longer connected; There are two solutions: Method 1: Disable the CRS service and then enable the CRS service. (Equivalent to restarting the system, because CRS is automatically started with the system), it must be executed on both nodes; Procedure: Node 1: 1. [root @ Rac1 ~] #/Etc/init. d/init. CRS stop Shutting down Oracle cluster ready Services (CRS ): Stopping resources. Successfully stopped CRS Resources Stopping cssd. Shutting down CSS daemon. Shutdown request successfully issued. Shutdown has begun. The daemons shoshould exit soon. 2. [root @ Rac1 ~] # Su-Oracle [Oracle @ Rac1 ~] $ Crs_stat-T CRS-0184: cannot communicate with the CRS daemon. 3. [Oracle @ Rac1 ~] $ Exit Logout [Root @ Rac1 ~] #/Etc/init. d/init. CRS start Startup will be queued to init within 90 seconds. 4. [root @ Rac1 ~] # Su-Oracle [Oracle @ Rac1 ~] $ Crs_stat-T Name type target State host ------------------------------------------------------------ Ora. ora9i. DB application online Rac1 Ora... i1.inst application online Rac1 Ora... i2.inst application online rac2 Ora... sm1.asm application online Rac1 Ora... c1.lsnr application online Rac1 Ora. rac1.gsd application online Rac1 Ora. rac1.ons application online Rac1 Ora. rac1.vip application online Rac1 Ora... sm2.asm application online rac2 Ora... c2.lsnr application online rac2 Ora. rac2.gsd application online rac2 Ora. rac2.ons application online rac2 Ora. rac2.vipApplication online rac2 [Oracle @ Rac1 ~] $   Node 2: 1. [root @ rac2 ~] #/Etc/init. d/init. CRS stop Shutting down Oracle cluster ready Services (CRS ): Stopping resources. Successfully stopped CRS Resources Stopping cssd. Shutting down CSS daemon. Shutdown request successfully issued. Shutdown has begun. The daemons shoshould exit soon. 2. [root @ rac2 ~] # Su-Oracle [Oracle @ rac2 ~] $ Crs_stat-T CRS-0184: cannot communicate with the CRS daemon. [Oracle @ rac2 ~] $ Exit Logout 3. [root @ rac2 ~] #/Etc/init. d/init. CRS start Startup will be queued to init within 90 seconds. 4. [root @ rac2 ~] # Su-Oracle [Oracle @ rac2 ~] $ Crs_stat-T Name type target State host ------------------------------------------------------------ Ora. ora9i. DB application online Rac1 Ora... i1.inst application online Rac1 Ora... i2.inst application online rac2 Ora... sm1.asm application online Rac1 Ora... c1.lsnr application online Rac1 Ora. rac1.gsd application online Rac1 Ora. rac1.ons application online Rac1 Ora. rac1.vip application online Rac1 Ora... sm2.asm application online rac2 Ora... c2.lsnr application online rac2 Ora. rac2.gsd application online rac2 Ora. rac2.ons application online rac2 Ora. rac2.vipApplication online rac2 [Oracle @ rac2 ~] $ Method 2: You can use crs_stop-F to forcibly stop the service and start the service (run on a node ); Procedure: [Oracle @ rac2 ~] $ Crs_stop-F ora. rac1.ons Attempting to stop 'ora. rac1.ons 'on member 'rac1' Stop of 'ora. rac1.ons 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac1.ons Attempting to start 'ora. rac1.ons 'on member 'rac1' Start of 'ora. rac1.ons 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac2.ons Attempting to stop 'ora. rac2.ons 'on member 'rac2' Stop of 'ora. rac2.ons 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac2.ons Attempting to start 'ora. rac2.ons 'on member 'rac2' Start of 'ora. rac2.ons 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac1.gsd Attempting to stop 'ora. rac1.gsd' on member 'rac1' Stop of 'ora. rac1.gsd' on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac1.gsd Attempting to start 'ora. rac1.gsd' on member 'rac1' Start of 'ora. rac1.gsd' on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac2.gsd Attempting to stop 'ora. rac2.gsd' on member 'rac2' Stop of 'ora. rac2.gsd' on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac2.gsd Attempting to start 'ora. rac2.gsd' on member 'rac2' Start of 'ora. rac2.gsd' on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac1.listener _ rac1.lsnr Attempting to stop 'ora. rac1.listener _ rac1.lsnr 'on member 'rac1' Stop of 'ora. rac1.listener _ rac1.lsnr 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac1.listener _ rac1.lsnr Attempting to start 'ora. rac1.listener _ rac1.lsnr 'on member 'rac1' Start of 'ora. rac1.listener _ rac1.lsnr 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac2.listener _ rac2.lsnr Attempting to stop 'ora. rac2.listener _ rac2.lsnr 'on member 'rac2' Stop of 'ora. rac2.listener _ rac2.lsnr 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac2.listener _ rac2.lsnr Attempting to start 'ora. rac2.listener _ rac2.lsnr 'on member 'rac2' Start of 'ora. rac2.listener _ rac2.lsnr 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac1.asm1. ASM Attempting to stop 'ora. rac1.asm1. ASM 'on member 'rac1' Stop of 'ora. rac1.asm1. ASM 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac1.asm1. ASM Attempting to start 'ora. rac1.asm1. ASM 'on member 'rac1' Start of 'ora. rac1.asm1. ASM 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. rac2.asm2. ASM Attempting to stop 'ora. rac2.asm2. ASM 'on member 'rac2' Stop of 'ora. rac2.asm2. ASM 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. rac2.asm2. ASM Attempting to start 'ora. rac2.asm2. ASM 'on member 'rac2' Start of 'ora. rac2.asm2. ASM 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. ora9i. ora9i1. Inst 'Ora. ora9i. ora9i1. Inst 'is already offline. CRS-0216: Cocould not stop resource 'ora. ora9i. ora9i1. inst '. [Oracle @ rac2 ~] $ Crs_start-F ora. ora9i. ora9i1. Inst Attempting to start 'ora. ora9i. ora9i1. Inst 'on member 'rac1' Start of 'ora. ora9i. ora9i1. Inst 'on member 'rac1' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. ora9i. ora9i2. Inst 'Ora. ora9i. ora9i2. Inst 'is already offline. CRS-0216: Cocould not stop resource 'ora. ora9i. ora9i2. inst '. [Oracle @ rac2 ~] $ Crs_start-F ora. ora9i. ora9i2. Inst Attempting to start 'ora. ora9i. ora9i2. Inst 'on member 'rac2' Start of 'ora. ora9i. ora9i2. Inst 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_stop-F ora. ora9i. DB Attempting to stop 'ora. ora9i. db' on member 'rac2' Stop of 'ora. ora9i. DB 'on member 'rac2' succeeded. [Oracle @ rac2 ~] $ Crs_start-F ora. ora9i. DB Attempting to start 'ora. ora9i. db' on member 'rac1' Start of 'ora. ora9i. DB 'on member 'rac1' succeeded.     [Oracle @ rac2 ~] $ Crs_stat-T Name type target State host ------------------------------------------------------------ Ora. ora9i. DB application online Rac1 Ora... i1.inst application online Rac1 Ora... i2.inst application online rac2 Ora... sm1.asm application online Rac1 Ora... c1.lsnr application online Rac1 Ora. rac1.gsd application online Rac1 Ora. rac1.ons application online Rac1 Ora. rac1.vip application online Rac1 Ora... sm2.asm application online rac2 Ora... c2.lsnr application online rac2 Ora. rac2.gsd application online rac2 Ora. rac2.ons application online rac2 Ora. rac2.vipApplication online rac2 Conclusion: In the RAC environment, if crs_stop-all is used to stop the RAC service, some services in the CRS may be in the unknown status, in this case, we can restart the CRS service in two ways: 1: convert to the root user using/etc/init. d/init. CRS stop first disables CRS and then uses/etc/init. d/init. CRS start is used to enable CRS. After CRS is enabled, a series of services of CRS are automatically started. Note that this method must be executed on both nodes; 2: Use the crs_stop-F parameter to turn off the services in the CRS state of unknown, and then use crs_start-F (with A-F parameter) to start all services. Reference: http://www.rachelp.nl/index_kb.php? Menu = articles & actie = show & id = 31 It must be executed on two nodes respectively; It is best to check the CRS, ASM, and database logs during the startup process: CRS log: [Oracle @ Rac1 ~] $ Tail-F/u01/APP/Oracle/product/10.2.0/crs_1/log/Rac1/alertrac1.log [Oracle @ rac2 ~] $ Tail-F/u01/APP/Oracle/product/10.2.0/crs_1/log/rac2/alertrac2.log ASM log: [Oracle @ Rac1 ~] $ Tail-F/u01/APP/Oracle/admin/+ ASM/bdump/Alert _ + asm1.log [Oracle @ rac2 ~] $ Tail-F/u01/APP/Oracle/admin/+ ASM/bdump/Alert _ + asm2.log Database Log: [Oracle @ Rac1 ~] $ Tail-F/u01/APP/Oracle/admin/ora9i/bdump/alert_ora9i1.log [Oracle @ rac2 ~] $ Tail-F/u01/APP/Oracle/admin/ora9i/bdump/alert_ora9i2.log