Only one node can be started after Oracle11gR2RAC is restarted

After the normal deployment of Oracle11gR2RAC is completed, perform the two-node restart operation to find that the cluster resources of one of the nodes cannot be started, and restart the cluster resources again.

After the normal deployment of Oracle 11gR2 RAC is completed, perform the two-node restart operation to find that the cluster resources of one of the nodes cannot be started, and restart the cluster resources again.

Problem Background:

After the normal deployment of Oracle 11gR2 RAC is complete, perform the two-node restart operation to find that the cluster resources of one of the nodes cannot be started, and restart the node that cannot start the cluster resources again. Restart the normal node and find that the faulty node resource is up. After the restart, the original normal node resource cannot be started.

Cluster Environment:

OS: RedHat performanise5.8 x86_x64

DB: Oracle EnterPrise Database 11.2.0.4.0 x86_x64

GRID: Oracle Grid Infrastructure 11.2.0.4 x86_x64

Both heartbeat and Internet NIC are bound to bond1;

Because the storage is from IBM, RDAC is used for multi-path aggregation;

Problem Analysis:

Check OS and cluster logs

Oracle 11g installation manual on RedHat Linux 5.8 _ x64 Platform

Installing Oracle 12C in Linux-6-64

Install Oracle 11gR2 (x64) in CentOS 6.4)

Steps for installing Oracle 11gR2 in vmwarevm

Install Oracle 11g XE R2 In Debian

Db01 grid log information

2014-09-29 15:36:36. 587:
[Ctssd (12616)] CRS-2405: The Cluster Time Synchronization Service on host db01 is shutdown by user
2014-09-29 15:36:36. 589:
[Mdnsd (6173)] CRS-5602: mDNS service stopping by request.
[Client (17411)] CRS-10001: 29-Sep-14 ACFS-9290: Waiting for ASM to shutdown.
2014-09-29 15:36:46. 395:
[Cssd (8958)] CRS-1603: CSSD on node db01 shutdown by user.
2014-09-29 15:36:46. 509:
[Ohasd (12463)] CRS-2767: Resource state recovery not attempted for 'ora.css dmonitor 'as its target state is OFFLINE
2014-09-29 15:36:46. 509:
[Ohasd (12463)] CRS-2769: Unable to failover resource 'ora.css dmonitor '.
2014-09-29 15:36:46. 608:
[Cssd (8958)] CRS-1660: The CSS daemon shutdown has completed

Ocssd
15:36:46. 052: [CSSD] [1122224448] clssnmSendingThread: sending status msg to all nodes
15:36:46. 053: [CSSD] [1122224448] clssnmSendingThread: sent 4 status msgs to all nodes
15:36:46. 388: [CSSD] [1089472832] clssgmExitGrock: client 1 (0x1261ee30), grock haip. cluster_interconnect, member 0
15:36:46. 388: [CSSD] [1089472832] clssgmUnregisterPrimary: Unregistering member 0 (0x1261ace0) in global grock haip. cluster_interconnect
15:36:46. 388: [CSSD] [1089472832] clssgmUnreferenceMember: global grock haip. cluster_interconnect member 0 refcount is 0
15:36:46. 388: [CSSD] [1089472832] clssgmAllocateRPCIndex: allocated rpc 357 (0x2aaaaaee8a58)
15:36:46. 388: [CSSD] [1089472832] clssgmRPC: rpc 0x2aaaaaee8a58 (RPC #357) tag (165002a) sent to node 1
15:36:46. 388: [CSSD] [1079531840] clssgmHandleMasterMemberExit: [s (1) d (1)]
15:36:46. 388: [CSSD] [1079531840] clssgmRemoveMember: grock haip. cluster_interconnect, member number 0 (0x1261ace0) node number 1 state 0x4 grock type 2
15:36:46. 388: [CSSD] [1079531840] clssgmGrantLocks: 0-> new master (1/2) group haip. cluster_interconnect
15:36:46. 388: [CSSD] [1079531840] clssgmRPCDone: rpc 0x2aaaaaee8a58 (RPC #357) state 6, flags 0x100
15:36:46. 388: [CSSD] [1079531840] clssgmDelMemCmpl: rpc 0x2aaaaaee8a58, ret 0, client 0x1261ee30 member 0x1261ace0
15:36:46. 388: [CSSD] [1079531840] clssgmFreeRPCIndex: freeing rpc 357
15:36:46. 388: [CSSD] [1079531840] clssgmAllocateRPCIndex: allocated rpc 358 (0x2aaaaaee8b00)
15:36:46. 388: [CSSD] [1089472832] clssgmDiscEndpcl: gipcDestroy 0x1a370
15:36:46. 388: [CSSD] [1079531840] clssgmRPCBroadcast: rpc (0x166002a), status (1), sendcount (1), filtered by specific properties:
15:36:46. 388: [CSSD] [1089472832] clssgmDeadProc: proc 0x1262a8a0
15:36:46. 388: [CSSD] [1089472832] clssgmDestroyProc: cleaning up proc (0x1262a8a0) con (0x1a341) skgpid ospid 12515 with 0 clients, refcount 0
15:36:46. 388: [CSSD] [1089472832] clssgmDiscEndpcl: gipcDestroy 0x1a341
15:36:46. 389: [CSSD] [1079531840] clssgmRPCDone: rpc 0x2aaaaaee8b00 (RPC #358) state 4, flags 0x402
15:36:46. 389: [CSSD] [1079531840] clssgmBroadcastGrockRcfgCmpl: RPC (0x166002a) of grock (haip. cluster_interconnect) received all acks, grock update sequence (4)
15:36:46. 389: [CSSD] [1079531840] clssgmFreeRPCIndex: freeing rpc 358
15:36:46. 395: [CSSD] [1089472832] clssgmExecuteClientRequest: MAINT recvd from proc 7 (0x1264c8e0)
15:36:46. 395: [CSSD] [1089472832] clssgmShutDown: stored ed explicit shutdown request from client.
15:36:46. 395: [CSSD] [1089472832] clssgmClientShutdown: total iocapables 0
15:36:46. 395: [CSSD] [1089472832] clssgmClientShutdown: graceful shutdown completed.
15:36:46. 395: [CSSD] [1089472832] clssgmClientShutdown: signaling to the agent that resource shocould remain down
15:36:46. 395: [CSSD] [1089472832] clssgmCompareSwapEventValue: changed CmInfo State val 0, from 11, changes 21
15:36:46. 395: [CSSD] [1079531840] clssgmPeerListener: terminating at incarn (307480783)
15:36:46. 395: [CSSD] [1079531840] clssgmPeerDeactivate: node 2 (db02), death 0, state 0x1 connstate 0xf
15:36:46. 395: [CSSD] [1079531840] clssgmCleanFuture: discarded 0 future msgs for 2
15:36:46. 395: [CSSD] [1079531840] clssgmDiscEndppl: gipcDestroy 0x26027
15:36:46. 496: [CSSD] [1089472832] clssnmSendManualShut: policying all nodes that this node has been manually shut down
15:36:46. 497: [GIPCHAUP] [1091049792] initialize: initiated discconnect umsg 0x12ac30d0 {msg 0x12ad9f08, ret listen (15), flags 0x2}, msg 0x12ad9f08 {type gipchaMsgTypeDisconnect (5 ), srcCid 00000000-00026038, dstCid 00000000-00000d5b}, endp 0x12ac12d0 [0000000000026038] {gipchaEndpoint: port '2017-da20-cc7e-b119 ', peer 'db02: gm2_db-cluster/Shanghai', srcCid 5996-00000000, dstCid 00000000-00000d5b, numSend 0, maxSend 100, groupListType 2, hagroup 0x12a9f4e0, usrFlags 0x4000, flags 0x21c}
15:36:46. 497: [GIPCHAUP] [1091049792] failed: completed DISCONNECT ret gipcretSuccess (0), umsg 0x12ac30d0 {msg 0x12ad9f08, ret gipcretSuccess (0), flags 0x2 }, msg 0x12ad9f08 {type gipchaMsgTypeDisconnect (5), srcCid 00000000-00026038, dstCid 00000000-00000d5b}, hendp 0x12ac12d0 [0000000000026038] {gipchaEndpoint: port '5996-da20-cc7e-b119 ', peer 'db02: gm2_db-cluster/comment ', srcCid 00000000-00026038, dstCid 00000000-00000d5b, numSend 0, maxSend 100, groupListType 2, hagroup 0x12a9f4e0, usrFlags 0x4000, flags 0x21c}
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x2aaaac8b3180) proc (0x2aaaac937fd0), iocapables 1.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x2aaaac937fd0), pid (6300), iocapables 1, client (0x2aaaac8b3180)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x2aaaac9773e0) proc (0x2aaaac937fd0), iocapables 2.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x2aaaac937fd0), pid (6300), iocapables 2, client (0x2aaaac9773e0)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x2aaaac958d50) proc (0x2aaaac99d570), iocapables 3.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x2aaaac99d570), pid (6208), iocapables 3, client (0x2aaaac958d50)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x12645650) proc (0x1264c8e0), iocapables 4.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x1264c8e0), pid (12518), iocapables 4, client (0x12645650)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x126171b0) proc (0x1264c8e0), iocapables 5.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x1264c8e0), pid (12518), iocapables 5, client (0x126171b0)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x1264cef0) proc (0x1264c8e0), iocapables 6.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x1264c8e0), pid (12518), iocapables 6, client (0x1264cef0)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x125f0340) proc (0x1260f910), iocapables 7.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x1260f910), pid (12520), iocapables 7, client (0x125f0340)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x125e98b0) proc (0x1260f910), iocapables 8.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x1260f910), pid (12520), iocapables 8, client (0x125e98b0)
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: Aborting client (0x12ac2790) proc (0x1260f910), iocapables 9.
15:36:46. 506: [CSSD] [1089472832] clssgmSendShutdown: I/O capable proc (0x1260f910), pid (12520), iocapables 9, client (0x12ac2790)
15:36:46. 506: [CSSD] [1089472832] clssgmClientShutdown: sending shutdown, fence_done 1
15:36:46. 608: [default] [1089472832] kgzf_fini: called