CRS-1006 , CRS-0215 故障案例

 安裝好sles 10 sp3 + Oracle 10g RAC之後，在配置監聽器時，總是提示主機bo2dbp上的監聽服務已經在運行，忽略錯誤之後手動在bo2dbp節點上啟，總是收到TNS-12545: Connect failed because target host or object does not exist錯誤資訊。後來發現節點bo2dbp的vip總是漂移到另一節點bo2dbs上，原來這才是罪魁禍首 ??????

1、叢集環境netca之後，節點bo2dbp監聽無法啟動

  手動嘗試啟動監聽仍不成功，查看了listener.ora以及監聽日誌未發現任何異常，查看主機/etc/hosts配置也正常

  無奈之下，刪除監聽器

  oracle@bo2dbp:~> crs_stat -t

  Name           Type           Target    State     Host      

  ------------------------------------------------------------

  ora....BP.lsnr application    ONLINE    OFFLINE              

  ora.bo2dbp.gsd application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbp.ons application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbp.vip application    ONLINE    ONLINE    bo2dbs    

  ora....BS.lsnr application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.gsd application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.ons application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.vip application    ONLINE    ONLINE    bo2dbs

2、節點bo2dbp.vip被漂移到bo2dbs

  oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_stat -t

  Name           Type           Target    State     Host      

  ------------------------------------------------------------

  ora.bo2dbp.gsd application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbp.ons application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbp.vip application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.gsd application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.ons application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.vip application    ONLINE    ONLINE    bo2dbs

3、嘗試為bo2dbp進行relocate也不聽使喚

  oracle@bo2dbp:~/product/10.2.0/crs_1/bin> ./crs_relocate ora.bo2dbp.vip -c bo2dbp

  Attempting to stop `ora.bo2dbp.vip` on member `bo2dbs`

  Stop of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

  Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`

  Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.

  Attempting to start `ora.bo2dbp.vip` on member `bo2dbs`

  Start of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

  CRS-0217: Could not relocate resource 'ora.bo2dbp.vip'.

4、crs_stop 出馬

  oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_stop ora.bo2dbp.vip

  Attempting to stop `ora.bo2dbp.vip` on member `bo2dbs`

  Stop of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

  oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_start ora.bo2dbp.vip

  Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`

  Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.

  Attempting to start `ora.bo2dbp.vip` on member `bo2dbs`

  Start of `ora.bo2dbp.vip` on member `bo2dbs` succeeded.

  我暈，又漂移到bo2dbs節點

5、狠一點，直接將節點2bo2dbs的crs停掉，這下沒得漂了吧 :)

  bo2dbs:/u01/app/oracle/product/10.2.0/crs_1/bin # ./crsctl stop crs

  Stopping resources. This could take several minutes.

  Successfully stopped CRS resources.

  Stopping CSSD.

  Shutting down CSS daemon.

  Shutdown request successfully issued.

6、露出馬腳

  oracle@bo2dbp:~/product/10.2.0/crs_1/bin> crs_start ora.bo2dbp.vip

  Attempting to start `ora.bo2dbp.vip` on member `bo2dbp`

  Start of `ora.bo2dbp.vip` on member `bo2dbp` failed.

  Remote start for `ora.bo2dbp.vip` failed on member `bo2dbs`

  CRS-1006: No more members to consider

  CRS-0215: Could not start resource 'ora.bo2dbp.vip'.

7、check log

  oracle@bo2dbp:~/product/10.2.0/crs_1/log/bo2dbp/racg> tail -50 ora.bo2dbp.vip.log

  2012-08-30 16:18:55.942: [  RACG][4250161648] [20571][4250161648][ora.bo2dbp.vip]: end for resource = ora.bo2dbp.vip, action = start, status = 1, time = 6.390s

  2012-08-30 16:19:30.968: [  RACG][4041843184] [20878][4041843184][ora.bo2dbp.vip]: checkIf: Default gateway is not defined (host=bo2dbp)

  Interface eth0 checked failed (host=bo2dbp)

  Invalid parameters, or failed to bring up VIP (host=bo2dbp)

  原來醉鬼禍首盡然是沒有預設網關，我倒倒倒......

  記得之前的安裝就碰到過在節點2上沒有預設網關導致vipca不能繼續的情形。這次又碰上了。汗。。。

  oracle@bo2dbp:~> cat /etc/sysconfig/network/routes

  cat: /etc/sysconfig/network/routes: No such file or directory

  oracle@bo2dbp:~> ssh bo2dbs cat /etc/sysconfig/network/routes

  default 192.168.7.254 - -

  使用root帳戶為其追加網關後，一切ok.

  oracle@bo2dbs:~> crs_stat -t

  Name           Type           Target    State     Host      

  ------------------------------------------------------------

  ora.bo2dbp.gsd application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbp.ons application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbp.vip application    ONLINE    ONLINE    bo2dbp    

  ora.bo2dbs.gsd application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.ons application    ONLINE    ONLINE    bo2dbs    

  ora.bo2dbs.vip application    ONLINE    ONLINE    bo2dbs  

8、問題環境

  sles 10/sp3 + Oracle 10g(10.2.0.4)

9、小結:

  a、細心細心再細心

  b、根據相應的日誌快速定位故障

oracle視頻教程請關注:http://u.youku.com/user_video/id_UMzAzMjkxMjE2.html