Oracle database failed to run root.sh in multi-CPU environment, ASM reported ORA-04031

Have friends and I have feedback that they're installing Linux 6.5 the 11.2.0.3 RAC appears to be abnormal, root.sh failed at the first node, requesting help

10:23

View the ASMCA log according to the record above

[Main] [2015-07-24 12:49:35.885 CST]  [sqlengine.reinitialize:738] Reinitializing SQLEngine ...
[Main] [2015-07-24 12:49:35.885 CST]  [oraclehome.getversion:889]  Oraclehome.getversion called. Current version:11.2.0.3.0
[Main] [2015-07-24 12:49:35.885 CST]  [oraclehome.getversion:957] Current Version from inventory:11.2.0.3.0
[Main] [2015-07-24 12:49:35.885 CST]  [oraclehome.getversion:889]  Oraclehome.getversion called. Current version:11.2.0.3.0
[Main] [2015-07-24 12:49:35.886 CST]  [oraclehome.getversion:957] Current Version from inventory:11.2.0.3.0
[Main] [2015-07-24 12:49:35.886 CST]  [oraclehome.getversion:889]  Oraclehome.getversion called. Current version:11.2.0.3.0
[Main] [2015-07-24 12:49:35.886 CST]  [oraclehome.getversion:957] Current Version from inventory:11.2.0.3.0
[Main] [2015-07-24 12:49:35.886 CST] [sqlplusengine.getcmmdparams:222] M_home 11.2.0.3.0
[Main] [2015-07-24 12:49:35.887 CST] [sqlplusengine.getcmmdparams:223] Version > 112 true
[Main] [2015-07-24 12:49:35.887 CST]  [sqlengine.getenvparams:555] Default Nls_lang:american_america. Al32utf8
[Main] [2015-07-24 12:49:35.887 CST]  [sqlengine.getenvparams:565] Nls_lang:american_america. Al32utf8
[Main] [2015-07-24 12:49:35.888 CST]  [sqlengine.initialize:325] Execing sqlplus/svrmgr Process ...
[Main] [2015-07-24 12:49:35.900 CST] [sqlengine.initialize:362] M_breaderstarted:false
[Main] [2015-07-24 12:49:35.900 CST]  [sqlengine.initialize:366] Starting Reader Thread ...
[Main] [2015-07-24 12:49:35.901 CST]  [sqlengine.initialize:415] Waiting for m_breaderstarted to is true
[Main] [2015-07-24 12:49:35.972 CST]  [sqlengine.done:2189] Done called
[Main] [2015-07-24 12:49:35.972 CST]  [usmcalogger.logexception:173] Severe:method Oracle.sysman.assistants.usmca.backend.USMInstance:configureLocalASM
[Main] [2015-07-24 12:49:35.973 CST]  [usmcalogger.logexception:174] Ora-01012:not logged on

[Main] [2015-07-24 12:49:35.973 CST] [usmcalogger.logexception:175] Oracle.sysman.assistants.util.sqlengine.sqlfatalerrorexception:ora-01012:not Logged on

Oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeImpl (sqlengine.java:1658)
Oracle.sysman.assistants.util.sqlEngine.SQLEngine.executeQuery (sqlengine.java:831)
Oracle.sysman.assistants.usmca.backend.USMInstance.configureLocalASM (usminstance.java:3036)
Oracle.sysman.assistants.usmca.service.UsmcaService.configureLocalASM (usmcaservice.java:1049)
Oracle.sysman.assistants.usmca.model.UsmcaModel.performConfigureLocalASM (usmcamodel.java:944)
Oracle.sysman.assistants.usmca.model.UsmcaModel.performOperation (usmcamodel.java:797)
Oracle.sysman.assistants.usmca.Usmca.execute (usmca.java:174)
Oracle.sysman.assistants.usmca.Usmca.main (usmca.java:369)
[Main] [2015-07-24 12:49:35.989 CST]  [usmcalogger.logexception:173] Severe:method Oracle.sysman.assistants.usmca.backend.USMInstance:configureLocalASM
[Main] [2015-07-24 12:49:35.989 CST]  [usmcalogger.logexception:174] Ora-03113:end-of-file on communication channel

[Main] [2015-07-24 12:49:35.989 CST] [usmcalogger.logexception:175] oracle.sysman.assistants.util.sqlengine.sqlfatalerrorexception:ora-03113: End-of-file on communication channel
As can be seen here, ASM instances cannot be logged (ORA-01012 and ORA-03113), and the ASM log is parsed based on such errors
Reconfiguration complete
Fri 24 12:49:29 2015
LCK0 started with pid=22, OS id=46913
Errors in File/u01/app/grid/diag/asm/+asm/+asm1/trace/+asm1_lmd0_46887.trc (incident=81):
ora-04031:unable to allocate 7072 bytes of shared memory ("Shared pool", "Unknown Object", "SGA Heap (1,1)", "GES resource")
Incident details in:/u01/app/grid/diag/asm/+asm/+asm1/incident/incdir_81/+asm1_lmd0_46887_i81.trc
Errors in File/u01/app/grid/diag/asm/+asm/+asm1/trace/+asm1_lck0_46913.trc (incident=177):
ora-04031:unable to allocate 760 bytes of shared memory ("Shared pool", "Unknown Object", "kkssp^1343", "KGLSS")
Incident details in:/u01/app/grid/diag/asm/+asm/+asm1/incident/incdir_177/+asm1_lck0_46913_i177.trc
Errors in File/u01/app/grid/diag/asm/+asm/+asm1/trace/+asm1_lmon_46885.trc (incident=73):
ora-04031:unable to allocate 632 bytes of shared memory ("Shared pool", "Unknown Object", "SGA Heap (1,1)", "Name-service")
Incident details in:/u01/app/grid/diag/asm/+asm/+asm1/incident/incdir_73/+asm1_lmon_46885_i73.trc
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
Errors in FILE/U01/APP/GRID/DIAG/ASM/+ASM/+ASM1/TRACE/+ASM1_LCK0_46913.TRC:
ora-04031:unable to allocate 760 bytes of shared memory ("Shared pool", "Unknown Object", "kkssp^1343", "KGLSS")
System State Dump requested by (Instance=1, osid=46913 (LCK0)), summary=[abnormal instance. Termination].
System state dumped to trace FILE/U01/APP/GRID/DIAG/ASM/+ASM/+ASM1/TRACE/+ASM1_DIAG_46879.TRC
Use Adrci or Support Workbench to package the incident.
411.1 at my Oracle Support for error and packaging details.
LCK0 (ospid:46913): Terminating the instance due to error 4031
Fri 24 12:49:35 2015
Ora-1092:opitsk aborting Process
Instance terminated by LCK0, PID = 46913
Further analysis of ASM logs is a familiar ORA-4031 problem with ASM, which means that the shared pool is not large enough when the database is executing root.sh using the default parameter file to start ASM (based on Oracle best practices, recommendations Memory_ Target=1536m and the above values), causing the problem to occur. Similar to bug 14292825 ORA-4031 in ASM as default memory parameters values for 11.2 ASM instances-low, fixed in 11.2.0.4 according to official description of the issue

Discover related default configuration via ASM log

Oracle Database 11g Enterprise Edition release 11.2.0.3.0-64bit Production
With the real application clusters and Automatic Storage Management options.
Oracle_home =/u01/app/11.2.0/grid
System Name:linux
Node NAME:RAC01
Release:2.6.32-358.el6.x86_64
Version: #1 SMP Tue 11:47:41 EST 2013
Machine:x86_64
Using parameter settings in Client-side Pfile/u01/app/11.2.0/grid/dbs/init+asm1.ora on machine RAC01
System parameters with Non-default values:
Large_pool_size = 16M
Instance_type = "ASM"
Remote_login_passwordfile= "EXCLUSIVE"
Asm_power_limit = 1
Diagnostic_dest = "/u01/app/grid"
Cluster communication is configured to with the following interface (s) for this instance
10.10.10.31
Cluster interconnect IPC version:oracle UDP/IP (generic)
IPC Vendor 1 Proto 2
Fri 24 12:49:27 2015
Check the number of CPUs by querying/proc/cpuinfo
processor:191
Vendor_id:genuineintel
CPU Family:6
model:62
Model Name:intel (R) Xeon (r) CPU e7-8850 v2 @ 2.30GHz
Stepping:7
CPU mhz:1200.000
Cache size:24576 KB
Physical Id:7
Siblings:24
Core Id:13
CPU Cores:12
apicid:251
Initial apicid:251
Fpu:yes
Fpu_exception:yes
CPUID level:13
Wp:yes
And is described in Subpools allocated During Startup (Doc ID 455179.1) under how-to determine the Default number of
Up to 7 Subpool (there are 192 CPUs, so the Subpool number is 7)



10:23

Minimum 512m memory per Suppool, so the shared pool requires 3.5G (the default value is hundreds of M, which is far from enough)

10:23

Due to the large number of CPUs, the subpools of shared pool is increased, making the shared pool more demanding. The reason for this failure can be summarized as follows:
With more CPUs, more shared pool is required, and due to less ASM default memory allocations in 11.2.0.3, there is a lack of shared pool at the time of ASM startup (its own default value is small, and shared pool demand is large). It is not surprising that the ORA-04031 occurs because the ASM does not start properly during the run of root.sh, which causes the root.sh to run unsuccessfully.
Treatment: Temporarily disable part of the CPU, then rerun root.sh, modify the ASM memory allocation, and then enable the CPU.
Special NOTE: This fault ACS brother met, so this time I can respond quickly, thank the ACs Brothers Help, and other authorized friends can look at: 3-10479952701 and 3-7976215751 SR description

Contact: Mobile Phone (13429648788) QQ (107644445)
Link: http://www.xifenfei.com/5985.html