Analysis and Handling of CRS CRS-2409 alarm information

Analysis and Handling of CRS-2409 alarm information problems reported by crs oracle 11.2.0.3 1. Check the CRS alert log of 1st nodes and find the following exception information 19:27:17. 609 [ctssd (18809056)] CRS-2409: The clock on host XXXdb1 is not synchronous with the mean cluster time. no action has been taken as the Cluster Time Synchronization Service is running in observer mode.2013-0X-XX 19:59:42. 312 [ctssd (18809056)] CRS-2409: The clock on host XXXdb1 is not synchronous with the mean cluster time. no action has been taken as the Cluster Time Synchronization Service is running in observer mode. the above error indicates that the CTSSD service of ORACLE is abnormal and cannot be in observation mode. 2. Error Message analysis 2.1. the NTPD service of the operating system is in startup state, so the CTSSD will not work, however, as long as the CTSSD service is started, it should normally be in observation mode 2.2. the NTPD service of the current OS is running, and, CTSSD cannot run in observation mode. 3. Check the time difference between the two nodes during troubleshooting.

XXXdb1:/u01/app/11.2.0.3/grid/log/XXXdb1$ssh XXXdb2 dateMon Jul 15 20:30:17 GMT+08:00 2013XXXdb1:/u01/app/11.2.0.3/grid/log/XXXdb1$dateMon Jul 15 20:30:18 GMT+08:00 2013

 

After check, there is no difference in time 3.2. Check the NTPD service of the OS

XXXdb1:/# lssrc -ls xntpd Program name:    /usr/sbin/xntpd Version:         3 Leap indicator:  00 (No leap second today.) Sys peer:        10.XXX.XXX.71 Sys stratum:     2 Sys precision:   -18 Debug/Tracing:   DISABLED Root distance:   0.000397 Root dispersion: 0.013458 Reference ID:    10.XXX.XXX.71 Reference time:  d58e6e41.d0fca000  Mon, Jul 15 2013 20:49:05.816 Broadcast delay: 0.003906 (sec) Auth delay:      0.000122 (sec) System flags:    bclient auth pll monitor filegen  System uptime:   30149695 (sec) Clock stability: 0.047607 (sec) Clock frequency: 0.000000 (sec) Peer: 10.XXX.XXX.71      flags: (configured)(sys peer)      stratum:  1, version: 3      our mode: client, his mode: serverSubsystem         Group            PID          Status xntpd            tcpip            4128900      active

 

After checking the two nodes, NTPD on the OS layer is running, and 3.3 time synchronization can be performed to check the running status of ctssd.

XXXdb1:/#su - gridXXXdb1:/home/grid$ crsctl stat res ora.ctssd -initNAME=ora.ctssdTYPE=ora.ctss.typeTARGET=ONLINESTATE=ONLINE on XXXdb1

 

After checking the two nodes, the CTSSD service has started 3.4 and the cause of the CTSS error is diagnosed using the CRS cluvfy tool.

XXXdb1:/home/grid$cluvfy comp clocksync -n all -verbose Verifying Clock Synchronization across the cluster nodes Checking if Clusterware is installed on all nodes...Check of Clusterware install passed Checking if CTSS Resource is running on all nodes...Check: CTSS Resource running on all nodes  Node Name                             Status                   ------------------------------------  ------------------------  XXXdb2                                passed                   XXXdb1                                passed                 Result: CTSS resource check passed Querying CTSS for time offset on all nodes...Result: Query of CTSS for time offset passed Check CTSS state started...Check: CTSS state  Node Name                             State                    ------------------------------------  ------------------------  XXXdb2                                Observer                 XXXdb1                                Observer               CTSS is in Observer state. Switching over to clock synchronization checks using NTP Starting Clock synchronization checks using Network Time Protocol(NTP)... NTP Configuration file check started...The NTP configuration file "/etc/ntp.conf" is available on all nodesNTP Configuration file check passed……Checking NTP daemon command line for slewing option "-x"Check: NTP daemon command line  Node Name                             Slewing Option Set?      ------------------------------------  ------------------------  XXXdb2                                no                       XXXdb1                                no                     Result:NTP daemon slewing option check failed on some nodesPRVF-5436 : The NTP daemon running on one or more nodes lacks the slewing option "-x"Result: Clock synchronization check using Network Time Protocol(NTP) failed

 

See the red font section above. During NTP slewingoption, both nodes fail because, NTP is not running in "-X" Mode 3.5. Check the configuration of NTPD on the OS layer (1) Check/etc/ntpd. conf

server 10.XXX.XXX.71broadcastclientdriftfile /etc/ntp.drifttracefile /etc/ntp.trace

 

(2) check whether the configuration of the/etc/rc. tcpip file has the following information:

# Start up Network Time Protocol (NTP) daemonstart /usr/sbin/xntpd "$src_running" -a "-x"

 

It seems that the configuration does not have a problem, but the current operation is not in the "-x" mode, it is likely that NTPD has been restarted, no "-x" parameter added at startup 4. problem handling method (executed on both nodes)

4.1 disable CTSSD (run by grid user) $ crsctl stat resource ora. ctssd-init4.2, stop NTPD (run by the root user) # stopsrc-s xntpd4.3, start NTPD, and add "-x" mode (run by the root user, it takes several minutes to start) # startsrc-s xntpd-a "-x" 4.4. start CTSSD (run by grid user) $ crsctl start resource ora. ctssd-init

 

5. Problem Solving verification 5.1. CTSD log record information after CTSSD is started

2013-XX-XX 19:18:00.427[ctssd(6357170)]CRS-2403:The Cluster Time Synchronization Service on host XXXdb1 is in observer mode.2013-XX-XX 19:18:00.888[ctssd(6357170)]CRS-2407:The new Cluster Time Synchronization Service reference node is host XXXdb2.2013-08-08 19:18:00.890[ctssd(6357170)]CRS-2401:The Cluster Time Synchronization Service started on host XXXdb1.

 

5.2 use the CRS cluvfy tool to verify whether the CTSS environment is normal

XXXdb1: // # su-gridXXXdb1:/home/grid $ cluvfy comp clocksync-n all-verbose ........ Check for NTP daemon or service using UDP port 123 Node Name Port Open? ------------------------------------ ------------------------ XXXdb2 yes XXXdb1 yes

Result: Clock synchronization check using Network Time Protocol (NTP) passedOracle Cluster Time Synchronization Services check passedVerification of Clock Synchronization within ss the cluster nodes was successful. from the above check information, we can see that all the previously incorrect items are normal now, and the check is successful.