Oracle RAC CSS 逾時計算 及 參數 misscount， Disktimeout 說明

 

一. 概述

在之前的文章：

       RAC 的一些概念性和原理性的知識

       http://blog.csdn.net/tianlesoftware/article/details/5331067

      

       提到OCSSD 這個進程是Clusterware最關鍵的進程，如果這個進程出現異常，會導致系統重啟，這個進程提供CSS(Cluster Synchronization Service)服務。 CSS 服務通過多種心跳機制即時監控叢集狀態，提供腦裂保護等基礎叢集服務功能。

       CSS 服務有2種心跳機制： 一種是通過私人網路的Network Heartbeat，另一種是通過Voting Disk的Disk Heartbeat.

       這2種心跳都有最大延時，對於Disk Heartbeat， 這個延時叫作IOT (I/O Timeout);對於Network Heartbeat, 這個延時叫MC(Misscount)。 這2個參數都以秒為單位，預設時IOT大於MC，在預設情況下，這2個參數是Oracle 自動判定的，並且不建議調整。

 

可以通過如下命令來查看參數值：

$crsctl get css disktimeout

$crsctl get css misscount

 

如：

[oracle@rac1 ~]$ crsctl get css disktimeout

200

[oracle@rac1 ~]$ crsctl get css misscount

60

這是這2個參數的預設值。

 

二. MOS 上相關的幾篇文章

How to start/stop the 10g CRS ClusterWare[ID 309542.1]

10g RAC: Steps To Increase CSS Misscount,Reboottime and Disktimeout [ID 284752.1]

CSS Timeout Computation in OracleClusterware [ID 294430.1]

RAC Assurance Support Team: RAC and OracleClusterware Starter Kit and Best Practices (Generic) [ID 810394.1]

 

2.1修改CSS Misscount 步驟：

  1)Shut down CRS on all but one node. For exact steps use Note 309542.1

  2)Execute crsctl as root to modify the misscount:

    $ORA_CRS_HOME/bin/crsctl set css misscount <n>

    where <n> is the maximum i/o latency to the voting disk +1 second

  3)Reboot the node where adjustment was made

  4)Start all other nodes shutdown in step 1

 

With the Patch:4896338 for 10.2.0.1 thereare two additional settings that can be tuned. This change is incorporated into the 10.2.0.2 and 10.1.0.6patchsets.  

 

 

These following are only relevant on10.2.0.1 with Patch:4896338，In addition to MissCount, CSS now has two more parameters:

  1)reboottime (default 3 seconds) - the amount of time allowed for a node  to complete a reboot after the CSS daemon hasbeen evicted. (I.E. how  long does ittake for the machine to completely shutdown when you do a reboot)

  2)disktimeout (default 200 seconds) - the maximum amount of time allowed      for a voting file I/O to complete; if thistime is exceeded the voting disk will be marked as offline.  Note that this is also the amount of timethat will be required for initial cluster formation, i.e. when no nodes havepreviously been up and in a cluster.

 

      $CRS_HOME/bin/crsctl set css reboottime <r> [-force]  (<r> is seconds)

      $CRS_HOME/bin/crsctl set css disktimeout <d> [-force] (<d>is seconds)

 

Confirm the new css  misscount setting via ocrdump

 

 

2.2 CSS Timeout Computation in OracleClusterware

2.2.1 MISSCOUNTDEFINITION AND DEFAULT VALUES
       The CSS misscount parameterrepresents the maximum time, in seconds, that a network heartbeat can be missedbefore entering into a cluster reconfiguration to evict the node. The followingare the default values for the misscount parameter and their respectiveversions when using Oracle Clusterware* in seconds:

 

OS	10g (R1 &R2)	11g
Linux	60	30
Unix	30	30
VMS	30	30
Windows	30	30

 

       *CSS misscount default value when using vendor (non-Oracle)clusterware is 600 seconds. This is to allow the vendor clusterwareample time to resolve any possible split brain scenarios.

       On AIX platforms with HACMP starting with 10.2.0.3 BP#1, themisscount is 30. This is documented in Note551658.1

 

2.2.2 CSS HEARTBEATMECHANISMS AND THEIR INTERRELATIONSHIP
       The synchronization servicescomponent (CSS) of the Oracle Clusterware maintains two heartbeat mechanisms

1.) the disk heartbeat to the voting deviceand

2.) the network heartbeat  across theinterconnect which establish and confirm valid node membership in the cluster.

       Bothof these heartbeat mechanisms have an associated timeout value. The diskheartbeat has an internal i/o timeout interval (DTO Disk TimeOut), in seconds,where an i/o to the voting disk must complete. The misscount parameter (MC), asstated above, is the maximum time, in seconds, that a network heartbeat can be missed. The disk heartbeat i/o timeout interval is directly related tothe misscount parameter setting. There has been some variation in thisrelationship 
between versions as described below:

9.x.x.x	NOTE, MISSCOUNT WAS A  DIFFERENT ENTITY IN THIS RELEASE
10.1.0.2	No one should be on this version
10.1.0.3	DTO = MC - 15 seconds
10.1.0.4	DTO = MC - 15 seconds
10.1.0.4+Unpublished Bug 3306964	DTO = MC - 3 seconds
10.1.0.4 with CRS II Merge patch	DTO =Disktimeout (Defaults to 200 seconds) Normally OR Misscount seconds only during initial Cluster formation or Slightly before reconfiguration
10.1.0.5	IOT = MC - 3 seconds
10.2.0.1 +Fix for unpublished Bug 4896338	IOT=Disktimeout (Defaults to 200 seconds) Normally OR Misscount seconds only during initial Cluster formation or Slightly before reconfiguration
10.2.0.2	Same as above (10.2.0.1 with Patch Bug:4896338
10.1 - 11.1	During node join and leave (reconfiguration) in a cluster we need to reconfigure, in that particular case we use Short Disk TimeOut (SDTO) which is in all versions SDTO = MC – reboottime (usually 3 seconds)

 

       Misscountdrives cluster membership reconfigurations and directly effects theavailability of the cluster. In most cases, the default settings for MC shouldbe acceptable.  Modifying the default value of misscount not onlyinfluences the timeout interval for the i/o to the voting disk, but alsoinfluences the tolerance for missed network heartbeats across the interconnect.

 

2.2.3 LONG LATENCIES TOTHE VOTING DISKS
       If I/O latencies to the voting diskare greater than the default DTO calculations noted above, the cluster mayexperience CSS node evictions depending on (a)the Oracle Clusterware (CRS)version, (b)whether merge patch has been applied and (c)the state of theCluster. More details on this are covered in the section "Change inBehavior with CRS Merge PATCH (4896338 on 10.2.0.1)".

       Theselatencies can be attributed to any number of problems in the i/o subsystem orproblems with any component in the i/o path. The following is a non exhaustivelist of reported problems which resulted in CSS node eviction due to latenciesto the voting disk longer than the default Oracle Clusterware i/o timeoutvalue(DTO):

1.       QLogic HBA cards with a LinkDown Timeout greater than the default misscount.

2.       Bad cables to the SAN/storagearray that effect i/o latencies

3.       SAN switch (like Brocade)failover latency greater than the default misscount

4.       EMC Clariion Array whentrespassing the SP to the backup SP greater than default misscount

5.       EMC PowerPath path errordetection and I/O repost and redirect greater than defaultmisscount  

6.       NetApp Cluster (CFO) failoverlatency greater than default misscount

7.       Sustained high CPU load whicheffects the CSSD disk ping monitoring thread

8.       Poor SAN network configurationthat creates latencies in the I/O path.

       The mostcommon problems relate to multi-path IO software drivers, and thereconfiguration times resulting from a failure in the IO path. Hardwareand (re)configuration issues that introduce these latencies should becorrected. Incompatible failover times with underlying OS, network or storagehardware or software may be addressed given a complete understanding of theconsiderations listed below.

 

       Misscount should NOT be modified to workaround theabove-mentioned issues. Oracle support recommends that you apply thelatest patchset which changes the CSS behaviour. More details covered innext section.

 

2.2.4 Change in Behaviorwith Bug:4896338 applied on top of 10.2.0.1
       Starting with 10.2.0.1+Bug:4896338,CSS will not evict the node from the cluster due to (DTO) I/O to voting disktaking more than misscount seconds unless it is during the initial clusterformation or slightly before reconfiguration. 
       So if we have a N number ofnodes in a cluster and one of the nodes takes more than misscountseconds to access the voting disk, the node will not be evicted as long asthe access to the voting disk is completed within disktimeoutseconds. Consequently with this patch, there is no need to increasethe misscount at all.

       Additionallythis merge patch introduces Disktimeout  which is the amount of time thata lack of disk ping to voting disk(s) will be tolerated.

 

Note:  applying the patch will notchange your value for Misscount.  

The table below explains inthe conditions under which the eviction will occur

 

Network Ping	Disk Ping	Reboot
Completes within misscount seconds	Completes within Misscount seconds	N
Completes within Misscount seconds	Takes more than misscount seconds but less than Disktimeout seconds	N
Completes within Misscount seconds	Takes more than Disktimeout seconds	Y
Takes more than Misscount Seconds	Completes within Misscount seconds	Y

 

 * By default Misscount is lessthan Disktimeout seconds

 

2.2.5 CONSIDERATIONS WHENCHANGING MISSCOUNT FROM THE DEFAULT VALUE

1.       Customers drive SLA and clusteravailability. The customer ultimately defines Service Levels and availabilityfor the cluster. Before recommending any change to misscount, the full impactof that change should be described and the impact to cluster availabilitymeasured.

2.       Customers may have timeout andretry logic in their applications. The impact of delaying reconfiguration maycause 'artificial' timeouts of the application, reconnect failures andsubsequent logon storms.

3.       Misscount timeout values areversion dependent and are subject to change. As we have seen, misscountcalculations are variable between releases and between versions within arelease. Creating a false dependency on misscount calculation in one versionmay not be appropriate for later versions.

4.       Internal I/O timeout interval(DTO) algorithms may change in later releases as stated above, there exists adirect relationship between the internal I/O timeout interval and misscount.This relationship is subject to change in later releases.

5.       An increase in misscount tocompensate for i/o latencies directly effects reconfiguration times for networkfailures. The network heartbeat is the primary indicator of connectivity withinthe cluster. Misscount is the tolerance level of missed 'check ins' thattrigger cluster reconfiguration. Increasing misscount will prolong the time totake corrective action in the event of network failure or other anomalieseffecting the availability of a node in the cluster. This directly effectscluster availability.

6.       Changing misscount toworkaround voting disk latencies will need to be corrected when the underlyingdisk latency is corrected, misscount needs to be set back to the default Thecustomer needs to document the change and set the parameter back to the defaultwhen the underlying storage I/O latency is resolved.

7.       Do not change default misscountvalues if you are  running Vendor Clusterware along with OracleClusterware. The default values for misscount should not be changed when usingvendor clusterware. Modifying misscount in this environment may causeclusterwide outages and potential corruptions.

8.       Changing misscount parameterincurs a clusterwide outage. As note below, the customer will need to schedule 
a clusterwide outage to make this change.

9.       Changing misscount should notbe used to compensate for poor configurations or faulty hardware

10.   Cluster and RDBMS availabilityare directly effected by high misscount settings.

11.   In case of stretched clustersand stretched storage systems and a site failure where we loose one storage andN number of nodes we go into a reconfiguration state and then we revert toShortDiskTimeOut value as internal I/O timeout for the votings. Several casesare known with stretched clusters where when a site failure happen the storagefailover cannot complete within SDTO. If the I/O to the votings is blocked morethan SDTO the result is node evictions on the surviving side. 

To Change MISSCOUNT back to default Please referto Note:284752.1
       THIS IS THE ONLY SUPPORTED METHOD.NOT FOLLOWING THIS METHOD RISKS EVICTIONS AND/OR CORRUPTING THE OCR

10g Release 2 MIRRORED VOTING DISKS AND VENDORMULTIPATHING SOLUTIONS 
       Oracle RAC 10g Release 2 allows formultiple voting disks so that  the customer does not have to rely on amultipathing solution from a storage vendor. You can have n voting disks (up to31) where n = m*2+1 where m is the number of disk failures you  want tosurvive. Oracle recommends each voting disk to be on a separate physical disk.

 

 

 

 

 

 

 

-------------------------------------------------------------------------------------------------------

Blog：http://blog.csdn.net/tianlesoftware

Weibo: http://weibo.com/tianlesoftware

Email: dvd.dba@gmail.com

DBA1 群：62697716(滿);   DBA2 群：62697977(滿)  DBA3 群：62697850(滿)  

DBA 超級群：63306533(滿);  DBA4 群： 83829929(滿) DBA5群： 142216823(滿) 

DBA6 群：158654907(滿)  聊天 群：40132017(滿)   聊天2群：69087192(滿)

--加群需要在備忘說明Oracle資料表空間和資料檔案的關係，否則拒絕申請