Oracle Tutorial: how to diagnose node restart Problems

Source: Internet
Author: User
Tags traceroute output
This article introduces how to diagnose node restart problems in the RAC environment. Applicable to 10gR2 and 11gR1. First, we will introduce the CRS process that can cause node restart. 1. ocssd:

This article introduces how to diagnose node restart problems in the RAC environment. Applicable to 10gR2 and 11gR1. First, we will introduce the CRS process that can cause node restart. 1. ocssd:

This article introduces how to diagnose node restart problems in the RAC environment. Applicable to 10gR2 and 11gR1.

First, we will introduce the CRS process that can cause node restart.
1. ocssd: its main functions are Node Monitoring and Group Management. It is one of the core processes of CRS. Node monitoring is used to monitor the health of nodes in a cluster. It is implemented through network heartbeat and disk heartbeat, if a node in the cluster continuously loses the disk heartbeat or network heartbeat, the node will be evicted from the cluster, that is, the node restarts. Node restart caused by group management is called node kill escalation (applicable only in 11gR1 and later versions). We will introduce it in detail in the following article. Restart must be completed within the specified time (reboot time, usually 3 seconds.

Network heartbeat: The ocssd. bin process sends network heartbeat information to each node in the cluster over the Intranet every second to confirm whether each node is normal. If a node continuously loses its network heartbeat and reaches the threshold value, misscount (30 seconds by default, and 600 seconds if other cluster management software exists) will vote by voting disk, the node that loses the network heartbeat is evicted from the cluster by the master node, that is, the node restarts. If the cluster contains only two nodes, split-brain occurs. As a result, nodes with a small node number survive, even nodes with a small node number have network problems.

Disk heartbeat: The ocssd. bin process registers the status information of the current node with all Voting disks (Voting File) every second. This process is called the disk heartbeat. If a node continuously loses the disk heartbeat to the threshold value, disk timeou (generally 200 seconds), the node will automatically restart to ensure cluster consistency. In addition, CRS only requires [N/2] + 1 voting disk available, where N is the number of voting disks, generally an odd number.

2. oclsomon: This process monitors whether ocssd is suspended. If ocssd. bin has performance problems, restart the node.
3. oprocd: This process only appears in Linux and Unix systems, and the third-party cluster management software is not installed. If the node is suspended, restart the node.

Note: all the preceding steps are generated by the script init.css d.

Next, we often collect information to diagnose node restart problems.
1. Operating System Logs
2. /Log/ <节点名称> /Cssd/ocssd. log
3. oprocd. log (/etc/Oracle/oprocd/*. log. * Or/var/opt/oracle/oprocd/*. log .*)
4. /Log/ <节点名称> /Cssd/oclsomon. log
5. Oracle OSWatcher report

Next we will discuss how to diagnose node restart.
1. Restart the node caused by ocssd.
If the following error occurs in ocssd. log, it indicates that the node is restarted because the network heartbeat is lost. Next, you need to view network-related information, such as operating system logs and OSW report (traceroute output) to determine whether there is a problem at the network layer (cluster interconnect) and determine the final cause.
[CSSD] 23:56:18. 749 [3086]> WARNING: clssnmPollingThread: node At 50% heartbeat fatal, eviction in 14.494 seconds
[CSSD] 23:56:25. 749 [3086]> WARNING: clssnmPollingThread: node At 75% heartbeat fatal, eviction in 7.494 seconds
[CSSD] 23:56:32. 749 [3086]> WARNING: clssnmPollingThread: node At 90% heartbeat fatal, eviction in 0.494 seconds
[CSSD] 23:56:33. 243 [3086]> TRACE: clssnmPollingThread: Eviction started for node , Flags 0x040d, state 3, wt4c 0
[CSSD] 23:56:33. 243 [3086]> TRACE: clssnmDiscHelper: , Node (4) connection failed, con (1128a5530), probe (0)
[CSSD] 23:56:33. 243 [3086]> TRACE: clssnmDiscHelper: node 4 clean up, con (1128a5530), init state 5, cur state 5
[CSSD] 23:56:33. 243 [3600]> TRACE: clssnmDoSyncUpdate: Initiating sync 196446491
[CSSD] 23:56:33. 243 [3600]> TRACE: clssnmDoSyncUpdate: diskTimeout set to (27000) MS

Note: If the time point for the above information appears in the ocssd. log of the master node is later than the restart time of the node, the reason for node restart is not the loss of network heartbeat.

If the following error occurs in ocssd. log, it indicates that the node is restarted because the disk heartbeat is lost. Next, you need to view the operating system logs and OSWatcher report (iostat output) to determine whether there is a problem at the I/o Level and determine the final cause.
18:34:37. 423: [CSSD] [150477728] clssnmvDiskOpen: Opening/dev/sdb8
18:34:37. 423: [CLSF] [150477728] Opened hdl: 0xf4336530 for dev:/dev/sdb8:
18:34:37. 429: [SKGFD] [150477728] ERROR:-9 (Error 27072, OS Error (Linux Error: 5: Input/output error
Additional information: 4
Additional information: 720913
Additional information:-1)
)
18:34:37. 429: [CSSD] [150477728] (: CSSNM00060:) clssnmvReadBlocks: read failed at offset 17 of/dev/sdb8
18:34:38. 205: [CSSD] [4110736288] (: CSSNM00058:) clssnmvDiskCheck: No I/O completions for 200880 MS for voting file/dev/sdb8)
18:34:38. 206: [CSSD] [4110736288] (: CSSNM00018:) clssnmvDiskCheck: Aborting, 0 of 1 configured voting disks available, need 1
18:34:38. 206: [CSSD] [4110736288] ################################## #
18:34:38. 206: [CSSD] [4110736288] clssscExit: CSSD aborting from thread clssnmvDiskPingMonitorThread
18:34:38. 206: [CSSD] [4110736288] ################################## #

2. node restart caused by oclsomon.
In oclsomon. if an error occurs in the log, it indicates that the node is restarted because the ocssd process is suspended. Because the ocssd process has a real-time (RT) priority, the operating system may have resource (such as cpu) competition, next, you need to view the operating system logs and OSW reports (output of vmstat and top) to determine the final cause.

3. Restart the node caused by oprocd.
If the following information appears in the oprocd log, it indicates that the restart of the node is caused by the oprocd process.

Dec 21 16:15:30. 369857 | LASTGASP | AlarmHandler: timeout (2312 msec) exceeds interval (1000 msec) + margin (500 msec). Rebooting NOW.

Because the oprocd process checks the system time to determine whether the operating system is suspended, correctly configuring ntp (or other time synchronization software) and adjusting diagwait = 13 can avoid node restart. In addition, if you need to modify the system time significantly, we recommend that you stop CRS and restart it after modification. Of course, we do not rule out that the oprocd node is restarted due to the suspension of the operating system. Therefore, you also need to check the OSWatcher report (output of vmstat and top) to determine the final cause.

This article only introduces the idea of diagnosing node restart problems, and needs to be used flexibly in actual problems.

For more information, read the following MOS article.
Note 265769.1: Troubleshooting 10g and 11.1 Clusterware Reboots
Note 1050693.1: Troubleshooting 11.2 Clusterware Node Evictions (Reboots)

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.