Analysis and Handling of CRS CRS-2409 alarm information

Oracle11.2.0.31. Check the CRSalertlog of node 1st with the error message. The following error message is displayed: 2013-0X-XX19: 27: 17.609 [ctssd (18809

Oracle 11.2.0.3 1. Check the CRS alert log of node 1st with the error message. The following error message is displayed: 2013-0X-XX 19:27:17. 609 [ctssd (18809

Oracle 11.2.0.3

1. error message

Check the CRS alert log On the 1st node and find the following exception information.

2013-0X-XX 19:27:17. 609

[Ctssd (18809056)] CRS-2409: The clock on host XXXdb1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

2013-0X-XX 19:59:42. 312

[Ctssd (18809056)] CRS-2409: The clock on host XXXdb1 is not synchronous with the mean cluster time. No action has been taken as the Cluster Time Synchronization Service is running in observer mode.

The preceding error indicates that the CTSSD service of ORACLE is abnormal and cannot be in observation mode.

2. error message Problem Analysis

2.1 If the NTPD service of the operating system is started, the CTSSD will not work, but as long as the CTSSD service is started, it should be in observation mode normally.

2.2 The NTPD service of the current OS is running, and the CTSSD cannot be running in observation mode.

3. troubleshooting Process

3.1 check whether the time of the two nodes is different

XXXdb1:/u01/app/11.2.0.3/grid/log/XXXdb1 $ ssh XXXdb2 date

Mon Jul 15 20:30:17 GMT + 2013

XXXdb1:/u01/app/11.2.0.3/grid/log/XXXdb1 $ date

Mon Jul 15 20:30:18 GMT + 2013

After check, there is no difference in time

3.2 check the NTPD service of the OS

XXXdb1: // # lssrc-ls xntpd

Program name:/usr/sbin/xntpd

Version: 3

Leap indicator: 00 (No leap second today .)

Sys peer: 10. XXX. XXX.71

Sys stratum: 2

Sys precision:-18

Debug/Tracing: DISABLED

Root distance: 0.000397

Root dispersion: 0.013458

Reference ID: 10. XXX. XXX.71

Reference time: d58e6e41. d0fca000 Mon, Jul 15 2013 20:49:05. 816

Broadcast delay: 0.003906 (sec)

Auth delay: 0.000122 (sec)

System flags: bclient auth pll monitor filegen

System uptime: 30149695 (sec)

Clock stability: 0.047607 (sec)

Clock frequency: 0.000000 (sec)

Peer: 10. XXX. XXX.71

Flags: (configured) (sys peer)

Stratum: 1, version: 3

Our mode: client, his mode: server

Subsystem Group PID Status

Xntpd tcpip 4128900 active

After checking the two nodes, NTPD on the OS layer is running and time synchronization can be performed.

3.3 check the running status of ctssd

XXXdb1:/# su-grid

XXXdb1:/home/grid $ crsctl stat res ora. ctssd-init

NAME = ora. ctssd

TYPE = ora. ctss. type

TARGET = ONLINE

STATE = ONLINE on XXXdb1

After checking the two nodes, the CTSSD service has been started.

3.4 use the CRS cluvfy tool to diagnose the cause of CTSS errors

XXXdb1:/home/grid $ cluvfy comp clocksync-n all-verbose

Verifying Clock Synchronization into ss the cluster nodes

Checking if Clusterware is installed on all nodes...

Check of Clusterware install passed

Checking if CTSS Resource is running on all nodes...

Check: CTSS Resource running on all nodes

Node Name Status

------------------------------------------------------------

XXXdb2 passed

XXXdb1 passed

Result: CTSS resource check passed

Querying CTSS for time offset on all nodes...

Result: Query of CTSS for time offset passed

Check CTSS state started...

Check: CTSS state

Node Name State

------------------------------------------------------------

XXXdb2 Observer

XXXdb1 Observer

CTSS is in Observer state. Switching over to clock synchronization checks using NTP

Starting Clock synchronization checks using Network Time Protocol (NTP )...

NTP Configuration file check started...

The NTP configuration file "/etc/ntp. conf" is available on all nodes

NTP Configuration file check passed

......

Checking NTP daemon command line for slewing option "-x"

Check: NTP daemon command line

Node Name Slewing Option Set?

------------------------------------------------------------

XXXdb2 no

XXXdb1 no

Result:

NTP daemon slewing option check failed on some nodes

The PRVF-5436: the NTP daemon running on one or more nodes lacks The slewing option "-x"

Result: Clock synchronization check using Network Time Protocol (NTP) failed

See the highlighted red font section above. During NTP slewingoption, both nodes fail because NTP is not running in "-X" mode.

3.5 check the configuration of NTPD on the OS layer

(1) Check/etc/ntpd. conf

Server 10. XXX. XXX.71

Broadcastclient

Driftfile/etc/ntp. drift

Tracefile/etc/ntp. trace

(2) Check the configuration of the/etc/rc. tcpip file.

The following information exists:

# Start up Network Time Protocol (NTP) daemon

Start/usr/sbin/xntpd "$ src_running"-a "-x"

It seems that the configuration is not correct, but the current operation is not in the "-x" mode. It is very likely that NTPD has been restarted and there is no "-x" parameter added at startup.

Next, let's take a look at the highlights of page 2nd:

Recommended reading:

Oracle 12C R1 RAC install CRS

CRS-0184/PRCR-1070 when Oracle 11g RAC executes root. sh

Oracle 10g R2 rac crs cannot be started, CRS-1604: CSSD voting file offline

Oracle RAC migration under VMware causes CRS-1006/CRS-0215/CRS-0233