Asm disk Group loading ORA-15183 error example
In Oracle 11gR2, ASM (Automatic Storage Management) is removed from the Database Component, and as an independent Component, it enters the scope of Grid Management.
This article mainly introduces a problem that occurs during the database startup and loading process. There are some differences with the policies recommended by the official MOS. Record it and keep it for future use.
1. Problem Description
The author's environment is Oracle Single Instance + Grid Infrastructure, and the version is 11.2.0.4. For security reasons, the latest security patch and upgrade patch are downloaded from MOS. The upgraded version is 11.2.0.4.6.
However, some problems have occurred in the last step of upgrading-executing SQL scripts.
SQL * Plus: Release 11.2.0.4.0 Production on Mon May 25 16:08:57 2015
Copyright (c) 1982,201 3, Oracle. All rights reserved.
SQL> conn/as sysdba
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 2087780352 bytes
Fixed Size 2254824 bytes
Variable Size 553650200 bytes
Database Buffers 1526726656 bytes
Redo Buffers 5148672 bytes
ORA-00205: error in identifying control file, check alert log for more info
From the perspective of prompt information, Oracle encountered a problem when locating the control file after the nomount phase was started.
To be honest, although it is a test environment, I am still panic. So I tried to use the srvctl cluster startup policy.
[Grid @ NCR-Standby-Asm ~] $ Srvctl start database-d sicsstb
PRCC-1014: sicsstb was already running.
PRCR-1004: Resource ora. sicsstb. db is already running
PRCR-1079: Failed to start resource ora. sicsstb. db
CRS-5702: Resource 'ora. sicsstb. db' is already running on 'NCR-standby-asm'
2. Problem Analysis
First, check whether the system can start with srvctl to determine the status of various resource resources on the GI.
[Grid @ NCR-Standby-Asm ~] $ Srvctl stop database-d sicsstb
[Grid @ NCR-Standby-Asm ~] $ Srvctl status asm
ASM is running on ncr-standby-asm
[Grid @ NCR-Standby-Asm ~] $ Crsctl stat res-t-init
--------------------------------------------------------------------------------
Name target state server STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
Ora. DATA. dg
ONLINE ncr-standby-asm
Ora. LISTENER. lsnr
ONLINE ncr-standby-asm
Ora. RECO. dg
ONLINE ncr-standby-asm
Ora. asm
ONLINE ncr-standby-asm Started
Ora. ons
OFFLINE ncr-standby-asm
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
Ora.css d
1 ONLINE ncr-standby-asm
Ora. diskmon
1 OFFLINE
Ora. evmd
1 ONLINE ncr-standby-asm
Ora. sicsstb. db
1 OFFLINE Instance Shutdown
[Grid @ NCR-Standby-Asm ~] $ Srvctl start database-d sicsstb
[Grid @ NCR-Standby-Asm ~] $
[Oracle @ NCR-Standby-Asm ~] $ Cd $ ORACLE_HOME/rdbms/admin
[Oracle @ NCR-Standby-Asm admin] $ sqlplus/nolog
SQL * Plus: Release 11.2.0.4.0 Production on Mon May 25 16:14:00 2015
Copyright (c) 1982,201 3, Oracle. All rights reserved.
SQL> conn/as sysdba
Connected.
SQL> select open_mode from v $ database;
OPEN_MODE
--------------------
READ WRITE
I guess this fault is related to ASM. Start with the database log (locate the failed start action) according to the concept of gradual exploitation ).
Mon May 25 16:09:28 2015
MMON started with pid = 17, OS id = 4151
Mon May 25 16:09:28 2015
MMNL started with pid = 18, OS id = 4153
Starting up 1 dispatcher (s) for network address '(ADDRESS = (PARTIAL = YES) (PROTOCOL = TCP ))'...
Starting up 1 shared server (s )...
NOTE: initiating MARK startup
Starting background process MARK
ORACLE_BASE from environment =/u02/app/oracle
Mon May 25 16:09:28 2015
MARK started with pid = 21, OS id = 4161
NOTE: MARK has subscribed
Mon May 25 16:09:28 2015
ALTER DATABASE MOUNT
Mon May 25 16:09:28 2015
Alter system set local_listener = '(ADDRESS = (PROTOCOL = TCP) (HOST = 127.0.0.1) (PORT = 1521)' SCOPE = memory sid = 'sicsstb ';
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library:/opt/oracle/extapi/64/asm/orcl/1/libasm. so
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library:/opt/oracle/extapi/64/asm/orcl/1/libasm. so
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup RECO was dismounted
ERROR: diskgroup RECO was not mounted
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+ RECO/sicsstb/controlfile/current.256.878897845'
ORA-17503: ksfdopn: 2 Failed to open file + RECO/sicsstb/controlfile/current.256.878897845
ORA-15001: diskgroup "RECO" does not exist or is not mounted.
ORA-15040: diskgroup is incomplete.
ORA-15040: diskgroup is incomplete.
ORA-00210: cannot open the specified control file
ORA-00202: control file: '+ DATA/sicsstb/controlfile/current.260.878897845'
ORA-17503: ksfdopn: 2 Failed to open file + DATA/sicsstb/controlfile/current.260.878897845
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-15040: diskgroup is incomplete.
ORA-15040: diskgroup is incomplete.
ORA-15040: diskgroup is incomplete.
ORA-205 signalled during: alter database mount...
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library:/opt/oracle/extapi/64/asm/orcl/1/libasm. so
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Mon May 25 16:09:31 2015
SUCCESS: diskgroup DATA was dismounted
ERROR: diskgroup DATA was not mounted
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
WARNING: FAILED to load library:/opt/oracle/extapi/64/asm/orcl/1/libasm. so
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
SUCCESS: diskgroup RECO was dismounted
ERROR: diskgroup RECO was not mounted
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-15183: ASMLIB initialization error [driver/agent not installed]
According to the prompts, during the mount stage, Oracle uses the control file location specified in spfile to access the + DATA and + RECO disk groups, but the two disk groups do not have the mount, so an error is reported.
Parameter, control file exists in ASM Diskgroup in image redundancy mode.
SQL> show parameter spfile
NAME TYPE VALUE
-----------------------------------------------------------------------------
Spfile string + DATA/sicsstb/spfilesicsstb. ora
SQL> show parameter control
NAME TYPE VALUE
-----------------------------------------------------------------------------
Control_file_record_keep_time integer 7
Control_files string + DATA/sicsstb/controlfile/curr
Ent.260.878897845, + RECO/sicsstb/controlfile/current.256.878
897845
Control_management_pack_access string DIAGNOSTIC + TUNING
Note: The ASM cannot be started here, but the ASM component is not started by the author. If it is simple because the ASM component is not enabled and the database service is started first, the error message is as follows:
[Oracle @ NCR-Standby-Asm ~] $ Sqlplus/nolog
SQL * Plus: Release 11.2.0.4.0 Production on Mon Jun 1 08:39:11 2015
Copyright (c) 1982,201 3, Oracle. All rights reserved.
SQL> conn/as sysdba
Connected to an idle instance.
SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+ DATA/sicsstb/spfilesicsstb. ora'
ORA-17503: ksfdopn: 10 Failed to open file + DATA/sicsstb/spfilesicsstb. ora
ORA-15077: cocould not locate ASM instance serving a required diskgroup
The nomount stage needs to access spfile. Our SPFILE is in + DATA. If ASM is really unavailable, it cannot even enter the nomount stage.
The prompt message seems to be a problem with the ASM driver. The operating system environment of the author is Red Hat Linux 6.5, using kmod as the ASM driver.
[Root @ NCR-Standby-Asm ~] # Rpm-qa | grep asm
Libatasmart-0.17-4.el6_2.x86_64
Oracleasmlib-2.0.4-1.el6.x86_64
Oracleasm-support-2.1.8-1.el6.x86_64
Kmod-oracleasm-2.0.6.rh1-3.el6_5.x86_64
Find the corresponding trace file, and you can see a more detailed description of the problem.
[Root @ NCR-Standby-Asm trace] # tail-n 200 sicsstb_rbal_4147.trc
Trace file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.4.0-64bit Production
With the Partitioning, Automatic Storage Management, OLAP, Data Mining
And Real Application Testing options
ORACLE_HOME =/u02/app/oracle/product/11.2.0/dbhome_1
System name: Linux
Node name: NCR-Standby-Asm
Release: 2.6.32-431. el6.x86 _ 64
Version: #1 SMP Sun Nov 10 22:19:54 EST 2013
Machine: x86_64
VM name: VMWare Version: 6
Instance name: sicsstb
Redo thread mounted by this instance: 0 <none>
Oracle process number: 15
Unix process pid: 4147, image: oracle @ NCR-Standby-Asm (RBAL)
* ** 16:09:31. 634
* ** Session id: (190.1) 16:09:31. 634
* ** Client id :() 16:09:31. 634
* ** Service name :() 16:09:31. 634
* ** Module name :() 16:09:31. 634
* ** Action name :() 16:09:31. 634
ERROR: asm_version error. err: driver/agent not installed rc: 2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc: 2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc: 2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc: 2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ERROR: asm_version error. err: driver/agent not installed rc: 2
ORA-15183: ASMLIB initialization error [driver/agent not installed]
ORA-15183: ASMLIB initialization error [driver/agent not installed]
Incident 9721 created, dump file:/u02/app/oracle/diag/rdbms/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], []
Error 488 detected in background process
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], []
Kjzduptcctx: policying DIAG for crash event
----- Abridged Call Stack Trace -----
Ksedsts () + 465 <-kjzdssdmp () + 267 <-kjzduptcctx () + 232 <-kjzdicrshnfy () + 63 <-ksuitm () + 5594 <-ksbrdp () + 3507 <-opirip () + 623 <-opidrv () + 603 <-sou2o () + 103 <-opimai_real () + 250 <-ssthrdmain () + 265 <-main () + 201 <-_ libc_start_main () + 253
----- End of Abridged Call Stack Trace -----
* ** 16:09:32. 865
RBAL (ospid: 4147): terminating the instance due to error 488
Ksuitm: waiting up to [5] seconds before killing DIAG (4129)
Terminate the process and view further details of the alert log.
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc (incident = 9721 ):
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], []
Incident details in:/u02/app/oracle/diag/rdbms/sicsstb/incident/incdir_9721/sicsstb_rbal_4147_i9721.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_rbal_4147.trc:
ORA-00600: internal error code, arguments: [kfdskAlloc0], [], [], [], [], [], [], [], [], [], []
RBAL (ospid: 4147): terminating the instance due to error 488
System state dump requested by (instance = 1, osid = 4147 (RBAL), summary = [abnormal instance termination].
System State dumped to trace file/u02/app/oracle/diag/rdbms/sicsstb/trace/sicsstb_diag_4129_20150525160933.trc
Dumping diagnostic data in directory = [cdmp_20150525160933], requested by (instance = 1, osid = 4147 (RBAL), summary = [abnormal instance termination].
Instance terminated by RBAL, pid = 4147
Mon May 25 16:09:40 2015
Adjusting the default value of parameter parallel_max_servers
From 160 to 120 due to the value of parameter processes (150)
The prompt message contains a ora-600 error and a half-tip half-suggestion message at the end, requiring that the number of parallel_max_servers parameters be increased. GI and ASM will be accompanied by multiple parallel working processes. I guess whether there may be too many processes and a fault caused by a small number of parameter settings.
Try to adjust the parameters.
SQL> show parameter parallel_max_servers
NAME TYPE VALUE
-----------------------------------------------------------------------------
Parallel_max_servers integer 120
SQL> alter system set parallel_max_servers = 150 scope = both;
System altered.
SQL> show parameter parallel_max
NAME TYPE VALUE
-----------------------------------------------------------------------------
Parallel_max_servers integer 150
On MOS, I also checked the permissions. Some articles in Oracle think it is a permission issue. However, there seems to be no too many problems.
[Oracle @ NCR-Standby-Asm ~] $ Cd $ ORACLE_HOME/bin
[Oracle @ NCR-Standby-Asm bin] $ ls-l grep oracle
Ls: cannot access grep: No such file or directory
-Rwsr-s -- x 1 oracle asmadmin 239882127 May 25 oracle
Then, restart the Database, and the service is normal.
SQL> startup
ORACLE instance started.
Total System Global Area 2087780352 bytes
Fixed Size 2254824 bytes
Variable Size 553650200 bytes
Database Buffers 1526726656 bytes
Redo Buffers 5148672 bytes
Database mounted.
Database opened.
Troubleshooting.
3. Conclusion
To be honest, I still have some questions about how to solve this fault. From an intuitive perspective, after the patch operation, the Oracle instance requires an increase in the number of processes, so some adjustments are required.