Home > Developer > Oracle

[RAC] Node 1 cannot be started due to lack of cssfatal

Source: Internet
Author: User
Developer on Alibaba Coud: Build your first app with APIs, SDKs, and tutorials on the Alibaba Cloud. Read more ＞

Environment:AIX5.3 + 10.2.0.5 RAC

Situation description:When RAC is disabled and restarted, one node cannot be started and the other node can be started normally.

Troubleshooting process:

1. Try to start the crs service of Node 1

Root #./init. crs start crs

2. Monitor crs logs during startup

OCSSD. log:

[CSSD] 09:27:54. 730> USER: Copyright 2014, Oracle version 10.2.0.5.0

[CSSD] 09:27:54. 730> USER: Starting CSS daemon on node nxjcdb1, number1, in cluster crs_dljc

[Clsdmt] Listening to (ADDRESS = (PROTOCOL = ipc) (KEY = nxjcdb1DBG_CSSD ))

[CSSD] 09:27:54. 790 [1]> TRACE: clssscmain: RT queuesetting: ON

[CSSD] 2014-01-16 09:27:55. 081 [1]> TRACE: clssscmain: local-only setto false

[CSSD] 09:27:55. 349 [1]> TRACE: clssnmReadNodeInfo: addednode 1 (nxjcdb1) to cluster

[CSSD] 09:27:55. 672 [1]> TRACE: clssnmReadNodeInfo: addednode 2 (nxjcdb2) to cluster

[CSSD] 09:27:55. 673 [1]> TRACE: clssnmInitNMInfo: Initialized with unique 1389835674

[CSSD] 09:27:55. 704 [1]> TRACE: clssNMInitialize: Initializing with OCR id (1516675067)

[CSSD] 09:27:55. 705 [1029]> TRACE: clssnm_skgxninit: HACMP clusterware detected

[CSSD] 09:27:56. 822 [1]> TRACE: clssnmNMInitialize: misscount set to (30)

[CSSD] 09:27:56. 900 [1]> TRACE: clssnmStartNM: reboottimeset to (3) sec

[CSSD] 09:27:56. 900 [1]> TRACE: clssnmNMInitialize: Networkheartbeat thresholds are: impending reconfig 15000 MS, reconfig start (misscount) 30000 MS

[CSSD] 09:27:57. 108 [1]> TRACE: clssnmDiskStateChange: statefrom 1 to 2 disk (0 // dev/rlvjc_voting)

[CSSD] 09:27:57. 108 [1030]> TRACE: clssnmvDPT: spawned for disk0 (/dev/rlvjc_voting)

[CSSD] 09:27:57. 146 [1030]> TRACE: clssnmvDiskOpen: Overwrotekill block for voting disk/dev/rlvjc_voting

[CSSD] 09:27:59. 163 [1030]> TRACE: clssnmDiskStateChange: statefrom 2 to 4 disk (0 // dev/rlvjc_voting)

[CSSD] 09:27:59. 164 [1]> ERROR: Internal Error Information:

Category: 1234

Operation: scls_scr_setval

Location: open

Other: cant open file

Dep: 2

[CSSD] 09:27:59. 164 [1]> ERROR: clsssclsfatal: failure 8 reading fatal mode

[CSSD] 09:27:59. 164 [1]> ERROR: ###################################

[CSSD] 09:27:59. 164 [1]> ERROR: clssscExit: CSSD abortingfrom thread Main

[CSSD] 09:27:59. 164 [1]> ERROR: ###################################

→According to the error message, it is preliminarily determined that the OCSSD cannot be started because Node 1 cannot voting disk.

[CSSD] --- dump grock state db ---

[CSSD] --- end of grock state dump ---

[CSSD] 09:27:59. 169 [1030]> TRACE: clssnmvReadDskHeartbeat: read ALL for Joining

[CSSD] 09:27:59. 169 [1030]> TRACE: clssnmvReadDskHeartbeat: node (2) is down. rcfg (2) wrtcnt (126947) LATS (1038806686) Disk lastSeqNo (126947)

[CSSD] ------- Begin Dump -------

[CSSD]

[CSSD]

[CSSD]

[CSSD]

[CSSD]

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863e0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100863f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086400 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086410 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086420 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086430 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086440 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086450 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086460 00 00 00 01 1008 61 98-00 00 00 01 10 c6 0b c0 ...... a .........

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086470 00 00 00 00 00 0000 01-00 00 00 00 02 00 03 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086480 00 00 00 01 1096 4a b0-00 00 00 00 00 00 00 00 ...... J .........

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086490 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 01 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864c0 00 00 00 00 0000 00 05-00 00 00 00 00 00 00 00 fa ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864e0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x1100864f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086500 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 166 [1]> TRACE: 0x110086510 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086520 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086530 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086540 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086550 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086560 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086570 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086580 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086590 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865a0 00 00 00 00 0000 00 00-00 00 00 0e 00 00 00 00 24 ............... $

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865e0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100865f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086600 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086610 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086620 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086630 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086640 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086650 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086660 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086670 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086680 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x110086690 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866b0 00 00 00 10 00 0000 00 97-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866c0 00 00 00 00 01 105d f6 10-00 00 00 00 01 10 95 ca 50 ......] ...... P

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866d0 00 00 00 01 1096 0a 70-00 00 00 00 3c 00 00 00 00 ....... p... <....

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866e0 00 00 00 01 1096 2a 90-00 00 00 01 00 00 00 01 ...... *.........

[CSSD] 09:28:00. 167 [1]> TRACE: 0x1100866f0 00 00 00 28 0000 00 00-00 00 00 01 10 00 16 08... (............

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086700 00 00 00 00 0000 00 00-00 00 00 01 10 4c 1a 90 ............. L ..

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086710 00 00 00 01 104c 1a 50-00 00 00 00 01 10 08 67 18 ..... l. P ...... g.

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086720 00 00 00 01 1008 67 18-00 00 00 00 00 00 00 00 00 ...... g .........

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086730 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086740 00 00 00 01 10cf 76 b0-00 00 00 00 00 00 00 00 ...... v .........

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086750 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086760 00 00 00 01 10cf 76 b0-00 00 00 00 00 00 00 00 ...... v .........

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086770 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086780 00 00 01 00 0000 00 00-00 00 01 10 bd f7 90 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086790 00 00 00 01 104c 17 50-00 00 00 00 00 00 00 00 00 ..... l. P ........

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867b0 00 00 00 00 01 1000 09 78-00 00 00 01 00 00 00 00 ....... x ........

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867c0 00 00 00 00 0000 00 01-00 00 00 01 01 10 3e df b0 ......> ..

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867d0 00 00 00 01 1046 49 90-00 00 00 01 10 46 4a 30 ..... FI ...... FJ0

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867e0 00 00 00 01 1046 4c 90-6e 78 6a 63 64 62 31 00 ...... FL. nxjcdb1.

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100867f0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086800 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086810 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086820 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086830 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086840 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086850 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086860 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086870 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086880 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x110086890 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 168 [1]> TRACE: 0x1100868e0 00 00 00 00 0000 00 00-6e 78 6a 63 64 62 31 2d ...... nxjcdb1-

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100868f0 70 72 69 00 0000 00 00 00-00 00 00 00 00 00 00 pri .............

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086900 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086910 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086920 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086930 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086940 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086950 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086960 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086970 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086980 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086990 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869a0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869b0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869c0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869d0 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869e0 00 00 00 0000 00 00-2f 6f 72 61 63 6c 65 2f ........ /oracle/

[CSSD] 09:28:00. 169 [1]> TRACE: 0x1100869f0 70 72 6f 64 7563 74 2f-31 30 2e 32 2e 30 2f 63 product/10.2.0/c

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a00 72 73 5f 31 0000 00 00 00-00 00 00 00 00 00 00 rs_1 ............

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a10 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a20 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a30 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a40 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a50 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a60 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a70 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a80 00 00 00 00 00 00 00 00-00 0000 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086a90 00 00 00 00 0000 00 00-00 00 00 00 00 00 00 00 ................

[CSSD] 09:28:00. 169 [1]> TRACE: 0x110086aa0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

/Dev/rlvjc_voting

3. log errors. Node 1's crs service cannot be started.

According to the error message, it is preliminarily determined that the OCSSD cannot be started because Node 1 cannot voting disk.

4. Problem locating

The cssfatal file is missing in the/etc/oracle/scls_scr/ballontt (host name)/oracle path. The file contains only one word "enable ".

~ Cat cssfatal

Enable

5. Problem Solving

Manually add this file

~ Vi cssfatal

Enable

6. The CRS of Node 1 is successfully started.

This article is an English version of an article which is originally in the Chinese language on aliyun.com and is provided for information purposes only. This website makes no representation or warranty of any kind, either expressed or implied, as to the accuracy, completeness ownership or reliability of the article or any translations thereof. If you have any concerns or complaints relating to the article, please send an email, providing a detailed description of the concern or complaint, to info-contact@alibabacloud.com. A staff member will contact you within 5 working days. Once verified, infringing content will be removed immediately.

Related Keywords:
win32 service cannot started convert due date to weeks catalyst control center cannot started windows 10 node 1 1 manual chest pain due to cervical spondylosis connection dropped due to connection reset es not responding due to long running script mean

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

What's Trending
Top 10 Tags
datastax versions naming convention zookeeper client class definition md5 microsoft sql server 2005 data structures exception handling error handling
Top 10 Keywords
microsoft download center down wordpress address url site address url wordpress address url windows installer 4 0 download 302 not found web address url definition site address url wordpress db2 integer mac os installation step by step pdf abbreviation for return

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

Get Started for Free

  • Sales Support

    1 on 1 presale consultation

    Chat Contact Sales

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    Open a Ticket

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.

    Learn More