ASM DISK Group載入ORA-15183錯誤一例
進入Oracle 11gR2，ASM（Automatic Storage Management）從Database組件中剝離出來，作為獨立組件Component進入Grid管理範疇。
本篇主要介紹筆者遇到的一個資料庫啟動載入過程中出現的問題。同官方MOS推薦的策略相比，有一些不同之處。記錄下來，留待需要的朋友待查使用。
1、問題說明
筆者環境是Oracle 單一實例+Grid Infrastructure，版本號碼為11.2.0.4。由於安全原因，從MOS上下載了最新的安全補丁和升級補丁。升級之後的版本為11.2.0.4.6。
但是，在升級最後步驟——執行SQL指令碼環節，出現了一些問題。
SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:08:57 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2087780352 bytes
Fixed Size 2254824 bytes
Variable Size 553650200 bytes
Database Buffers 1526726656 bytes
Redo Buffers 5148672 bytes
ORA-00205: error in identifying control file, check alert log for more info
從提示資訊角度看，Oracle在經曆啟動nomount階段之後，在定位control file的過程中出現了問題。
老實說，雖然是測試環境，但是筆者還是比較驚慌的。於是嘗試使用srvctl叢集件啟動策略。
[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb
PRCC-1014 : sicsstb was already running
PRCR-1004 : Resource ora.sicsstb.db is already running
PRCR-1079 : Failed to start resource ora.sicsstb.db
CRS-5702: Resource 'ora.sicsstb.db' is already running on 'ncr-standby-asm'
2、問題分析
首先確認系統是否可以使用srvctl啟動，判斷一下GI上面各種資源resource狀態。
[grid@NCR-Standby-Asm ~]$ srvctl stop database -d sicsstb
[grid@NCR-Standby-Asm ~]$ srvctl status asm
ASM is running on ncr-standby-asm
[grid@NCR-Standby-Asm ~]$ crsctl stat res -t -init
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE ncr-standby-asm
ora.LISTENER.lsnr
ONLINE ONLINE ncr-standby-asm
ora.RECO.dg
ONLINE ONLINE ncr-standby-asm
ora.asm
ONLINE ONLINE ncr-standby-asm Started
ora.ons
OFFLINE OFFLINE ncr-standby-asm
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
1 ONLINE ONLINE ncr-standby-asm
ora.diskmon
1 OFFLINE OFFLINE
ora.evmd
1 ONLINE ONLINE ncr-standby-asm
ora.sicsstb.db
1 OFFLINE OFFLINE Instance Shutdown
[grid@NCR-Standby-Asm ~]$ srvctl start database -d sicsstb
[grid@NCR-Standby-Asm ~]$
[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/rdbms/admin
[oracle@NCR-Standby-Asm admin]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.4.0 Production on Mon May 25 16:14:00 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected.
SQL> select open_mode from v$database;
OPEN_MODE
--------------------
READ WRITE
筆者猜測，這個故障和ASM相關。按照逐步抽絲剝繭的思路，先從資料庫日誌入手（找到失敗啟動的那次動作）。
Mon May 25 16:09:28 2015
MMON started with pid=17, OS id=4151
Mon May 25 16:09:28 2015
MMNL started with pid=18, OS id=4153
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
starting up 1 shared server(s) ...
NOTE: initiating MARK startup
Starting background process MARK
ORACLE_BASE from environment = /u02/app/oracle
Mon May 25 16:09:28 2015
MARK started with pid=21, OS id=4161
NOTE: MARK has subscribed
Mon May 25 16:09:28 2015
ALTER DATABASE MOUNT
Mon May 25 16:09:28 2015
ALTER SYSTEM SET local_listener=' (ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=1521))' SCOPE=MEMORY SID='sicsstb';
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup RECO was dismounted
ERROR: diskgroup RECO was not mounted
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+RECO/sicsstb/controlfile/current.256.878897845'
ORA-17503: ksfdopn:2 Failed to open file +RECO/sicsstb/controlfile/current.256.878897845
ORA-15001: diskgroup "RECO" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+DATA/sicsstb/controlfile/current.260.878897845'
ORA-17503: ksfdopn:2 Failed to open file +DATA/sicsstb/controlfile/current.260.878897845
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-15040: diskgroup is incomplete
ORA-205 signalled during: ALTER DATABASE MOUNT...
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Mon May 25 16:09:31 2015
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library: /opt/oracle/extapi/64/asm/orcl/1/libasm.so
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup RECO was dismounted
ERROR: diskgroup RECO was not mounted
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
從提示資訊看，Oracle在mount階段時候，利用spfile中指定的control file位置去訪問+DATA和+RECO磁碟組，但是兩個磁碟組沒有mount，所以才開始報錯。
參數中，control file以鏡像冗餘方式存在在ASM Diskgroup中。
SQL> show parameter spfile
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
spfile string +DATA/sicsstb/spfilesicsstb.ora
SQL> show parameter control
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
control_file_record_keep_time integer 7
control_files string +DATA/sicsstb/controlfile/curr
ent.260.878897845, +RECO/sicsstb/controlfile/current.256.878
897845
control_management_pack_access string DIAGNOSTIC+TUNING
注意：此處的ASM無法啟動，並不是筆者沒有啟動ASM組件。如果是簡單因為ASM組件沒有開啟，先啟動資料庫服務的話，錯誤資訊如下：
[oracle@NCR-Standby-Asm ~]$ sqlplus /nolog
SQL*Plus: Release 11.2.0.4.0 Production on Mon Jun 1 08:39:11 2015
Copyright (c) 1982, 2013, Oracle. All rights reserved.
SQL> conn / as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/sicsstb/spfilesicsstb.ora'
ORA-17503: ksfdopn:10 Failed to open file +DATA/sicsstb/spfilesicsstb.ora
ORA-15077: could not locate ASM instance serving a required diskgroup
nomount階段要訪問spfile，我們的SPFILE是在+DATA裡面，如果ASM真的停用話，連nomount階段都不能進入。
提示資訊上，似乎是筆者的ASM驅動有問題。筆者作業系統環境是Red Hat Linux 6.5，使用kmod作為ASM驅動程式。
[root@NCR-Standby-Asm ~]# rpm -qa | grep asm
libatasmart-0.17-4.el6_2.x86_64
oracleasmlib-2.0.4-1.el6.x86_64
oracleasm-support-2.1.8-1.el6.x86_64
kmod-oracleasm-2.0.6.rh1-3.el6_5.x86_64
尋找對應產生的trace檔案，可以看到問題的更詳細描述。
[root@NCR-Standby-Asm trace]# tail -n 200 sicsstb_rbal_4147.trc
Trace file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
and Real Application Testing options
ORACLE_HOME = /u02/app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: NCR-Standby-Asm
Release: 2.6.32-431.el6.x86_64
Version: #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine: x86_64
VM name: VMWare Version: 6
Instance name: sicsstb
Redo thread mounted by this instance: 0 <none>
Oracle process number: 15
Unix process pid: 4147, image: oracle@NCR-Standby-Asm (RBAL)
*** 2015-05-25 16:09:31.634
*** SESSION ID:(190.1) 2015-05-25 16:09:31.634
*** CLIENT ID:() 2015-05-25 16:09:31.634
*** SERVICE NAME:() 2015-05-25 16:09:31.634
*** MODULE NAME:() 2015-05-25 16:09:31.634
*** ACTION NAME:() 2015-05-25 16:09:31.634
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc:2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Incident 9721 created, dump file: /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
error 488 detected in background process
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
kjzduptcctx: Notifying DIAG for crash event
----- Abridged Call Stack Trace -----
ksedsts()+465<-kjzdssdmp()+267<-kjzduptcctx()+232<-kjzdicrshnfy()+63<-ksuitm()+5594<-ksbrdp()+3507<-opirip()+623<-opidrv()+603<-sou2o()+103<-opimai_real()+250<-ssthrdmain()+265<-main()+201<-__libc_start_main()+253
----- End of Abridged Call Stack Trace -----
*** 2015-05-25 16:09:32.865
RBAL (ospid: 4147): terminating the instance due to error 488
ksuitm: waiting up to [5] seconds before killing DIAG(4129)
終止進程操作，查看alert log的進一步詳細資料。
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc (incident=9721):
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
Incident details in: /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], [], []
RBAL (ospid: 4147): terminating the instance due to error 488
System state dump requested by (instance=1, osid=4147 (RBAL)), summary=[abnormal instance termination].
System State dumped to trace file /u02/app/oracle/diag/rdbms/sicsstb/sicsstb/trace/sicsstb_diag_4129_20150525160933.trc
Dumping diagnostic data in directory=[cdmp_20150525160933], requested by (instance=1, osid=4147 (RBAL)), summary=[abnormal instance termination].
Instance terminated by RBAL, pid = 4147
Mon May 25 16:09:40 2015
Adjusting the default value of parameter parallel_max_servers
from 160 to 120 due to the value of parameter processes (150)
提示資訊中出現了ora-600錯誤，並且在最後有一個半提示半建議的資訊，要求提升參數parallel_max_servers的數量參數。GI和ASM要伴隨著多個並行背景工作處理序，筆者猜測一種可能是不是進程數量過多，參數設定較小而引起的故障。
嘗試將參數進行調整。
SQL> show parameter parallel_max_servers
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
parallel_max_servers integer 120
SQL> alter system set parallel_max_servers=150 scope=both;
System altered.
SQL> show parameter parallel_max
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
parallel_max_servers integer 150
在MOS上，筆者也進行了檢查，Oracle一些文章認為是許可權問題。但是似乎沒有過多問題。
[oracle@NCR-Standby-Asm ~]$ cd $ORACLE_HOME/bin
[oracle@NCR-Standby-Asm bin]$ ls -l grep oracle
ls: cannot access grep: No such file or directory
-rwsr-s--x 1 oracle asmadmin 239882127 May 25 17:06 oracle
之後，重啟Database，服務正常。
SQL> startup
ORACLE instance started.
Total System Global Area 2087780352 bytes
Fixed Size 2254824 bytes
Variable Size 553650200 bytes
Database Buffers 1526726656 bytes
Redo Buffers 5148672 bytes
Database mounted.
Database opened.
故障解決。
3、結論
老實說，筆者對這個故障的解決還是有一些不明白的地方。從直觀看，在進行補丁操作之後，Oracle執行個體對進程數目要求是增加的，所以需要進行一些調整。
Start building with 50+ products and up to 12 months usage for Elastic Compute Service
1 on 1 presale consultation
24/7 Technical Support 6 Free Tickets per Quarter Faster Response
Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.
A comprehensive suite of global cloud computing services to power your business
Payment Methods We Support
