Environment:AIX5.3 + 10.2.0.5 RAC
Situation description:When RAC is disabled and restarted, one node cannot be started and the other node can be started normally.
Troubleshooting process:
1. Try to start the crs service of Node 1
Root #./init. crs start crs
2. Monitor crs logs during startup
OCSSD. log:
[CSSD] 09:27:54. 730> USER: Copyright 2014, Oracle version 10.2.0.5.0
[CSSD] 09:27:54. 730> USER: Starting CSS daemon on node nxjcdb1, number1, in cluster crs_dljc
[Clsdmt] Listening to (ADDRESS = (PROTOCOL = ipc) (KEY = nxjcdb1DBG_CSSD ))
[CSSD] 09:27:54. 790 [1]> TRACE: clssscmain: RT queuesetting: ON
[CSSD] 2014-01-16 09:27:55. 081 [1]> TRACE: clssscmain: local-only setto false
[CSSD] 09:27:55. 349 [1]> TRACE: clssnmReadNodeInfo: addednode 1 (nxjcdb1) to cluster
[CSSD] 09:27:55. 672 [1]> TRACE: clssnmReadNodeInfo: addednode 2 (nxjcdb2) to cluster
[CSSD] 09:27:55. 673 [1]> TRACE: clssnmInitNMInfo: Initialized with unique 1389835674
[CSSD] 09:27:55. 704 [1]> TRACE: clssNMInitialize: Initializing with OCR id (1516675067)
[CSSD] 09:27:55. 705 [1029]> TRACE: clssnm_skgxninit: HACMP clusterware detected
[CSSD] 09:27:56. 822 [1]> TRACE: clssnmNMInitialize: misscount set to (30)
[CSSD] 09:27:56. 900 [1]> TRACE: clssnmStartNM: reboottimeset to (3) sec
[CSSD] 09:27:56. 900 [1]> TRACE: clssnmNMInitialize: Networkheartbeat thresholds are: impending reconfig 15000 MS, reconfig start (misscount) 30000 MS
[CSSD] 09:27:57. 108 [1]> TRACE: clssnmDiskStateChange: statefrom 1 to 2 disk (0 // dev/rlvjc_voting)
[CSSD] 09:27:57. 108 [1030]> TRACE: clssnmvDPT: spawned for disk0 (/dev/rlvjc_voting)
[CSSD] 09:27:57. 146 [1030]> TRACE: clssnmvDiskOpen: Overwrotekill block for voting disk/dev/rlvjc_voting
[CSSD] 09:27:59. 163 [1030]> TRACE: clssnmDiskStateChange: statefrom 2 to 4 disk (0 // dev/rlvjc_voting)
[CSSD] 09:27:59. 164 [1]> ERROR: Internal Error Information:
Category: 1234
Operation: scls_scr_setval
Location: open
Other: cant open file
Dep: 2
[CSSD] 09:27:59. 164 [1]> ERROR: clsssclsfatal: failure 8 reading fatal mode
[CSSD] 09:27:59. 164 [1]> ERROR: ###################################
[CSSD] 09:27:59. 164 [1]> ERROR: clssscExit: CSSD abortingfrom thread Main
[CSSD] 09:27:59. 164 [1]> ERROR: ###################################
→According to the error message, it is preliminarily determined that the OCSSD cannot be started because Node 1 cannot voting disk.
[CSSD] --- dump grock state db ---
[CSSD] --- end of grock state dump ---
[CSSD] 09:27:59. 169 [1030]> TRACE: clssnmvReadDskHeartbeat: read ALL for Joining
[CSSD] 09:27:59. 169 [1030]> TRACE: clssnmvReadDskHeartbeat: node (2) is down. rcfg (2) wrtcnt (126947) LATS (1038806686) Disk lastSeqNo (126947)
[CSSD] ------- Begin Dump -------
[CSSD]
[CSSD]
[CSSD]
[CSSD]
[CSSD]
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863e0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086400 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086410 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086420 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086430 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086440 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086450 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086460 00 00 00 01 1008 61 98-00 00 00 01 10 c6 0b c0 ...... a .........
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086470 00 00 00 00 00 0000 01-00 00 00 00 02 00 03 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086480 00 00 00 01 1096 4a b0-00 00 00 00 00 00 00 00 ...... J .........
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086490 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 01 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864c0 00 00 00 00 0000 00 05-00 00 00 00 00 00 00 00 fa ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864e0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086500 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086510 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086520 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086530 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086540 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086550 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086560 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086570 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086580 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086590 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865a0 00 00 00 00 0000 00 00-00 00 00 0e 00 00 00 00 24 ............... $
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865e0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086600 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086610 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086620 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086630 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086640 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086650 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086660 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086670 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086680 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086690 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866b0 00 00 00 10 00 0000 00 97-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866c0 00 00 00 00 01 105d f6 10-00 00 00 00 01 10 95 ca 50 ......] ...... P
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866d0 00 00 00 01 1096 0a 70-00 00 00 00 3c 00 00 00 00 ....... p... <....
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866e0 00 00 00 01 1096 2a 90-00 00 00 01 00 00 00 01 ...... *.........
[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866f0 00 00 00 28 0000 00 00-00 00 00 01 10 00 16 08... (............
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086700 00 00 00 00 0000 00 00-00 00 00 01 10 4c 1a 90 ............. L ..
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086710 00 00 00 01 104c 1a 50-00 00 00 00 01 10 08 67 18 ..... l. P ...... g.
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086720 00 00 00 01 1008 67 18-00 00 00 00 00 00 00 00 00 ...... g .........
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086730 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086740 00 00 00 01 10cf 76 b0-00 00 00 00 00 00 00 00 ...... v .........
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086750 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086760 00 00 00 01 10cf 76 b0-00 00 00 00 00 00 00 00 ...... v .........
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086770 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086780 00 00 01 00 0000 00 00-00 00 01 10 bd f7 90 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086790 00 00 00 01 104c 17 50-00 00 00 00 00 00 00 00 00 ..... l. P ........
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867b0 00 00 00 00 01 1000 09 78-00 00 00 01 00 00 00 00 ....... x ........
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867c0 00 00 00 00 0000 00 01-00 00 00 01 01 10 3e df b0 ......> ..
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867d0 00 00 00 01 1046 49 90-00 00 00 01 10 46 4a 30 ..... FI ...... FJ0
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867e0 00 00 00 01 1046 4c 90-6e 78 6a 63 64 62 31 00 ...... FL. nxjcdb1.
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086800 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086810 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086820 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086830 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086840 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086850 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086860 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086870 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086880 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086890 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868e0 00 00 00 00 0000 00 00-6e 78 6a 63 64 62 31 2d ...... nxjcdb1-
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100868f0 70 72 69 00 0000 00 00 00-00 00 00 00 00 00 00 pri .............
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086900 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086910 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086920 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086930 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086940 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086950 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086960 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086970 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086980 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086990 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869e0 00 00 00 0000 00 00-2f 6f 72 61 63 6c 65 2f ........ /oracle/
[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869f0 70 72 6f 64 7563 74 2f-31 30 2e 32 2e 30 2f 63 product/10.2.0/c
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a00 72 73 5f 31 0000 00 00 00-00 00 00 00 00 00 00 rs_1 ............
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a10 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a20 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a30 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a40 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a50 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a60 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a70 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a80 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a90 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................
[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086aa0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
/Dev/rlvjc_voting
3. log errors. Node 1's crs service cannot be started.
According to the error message, it is preliminarily determined that the OCSSD cannot be started because Node 1 cannot voting disk.
4. Problem locating
The cssfatal file is missing in the/etc/oracle/scls_scr/ballontt (host name)/oracle path. The file contains only one word "enable ".
~ Cat cssfatal
Enable
5. Problem Solving
Manually add this file
~ Vi cssfatal
Enable
6. The CRS of Node 1 is successfully started.