11.2.0.3ASM instance ORA-4031 error causes database archive failure

11.2.0.3ASM instance ORA-4031 error causes database archive failure

11.2.0.3 the database archiving failed due to a ORA-4031 error in the ASM instance

Environment:
Platform: RedHat EnterPrise 5.8 X86_X64
Database: Oracle EnterPrise 11.2.0.3
Cluster software: Oracle grid 11.2.0.3

Fault symptom:
An archive failure occurs in the database, and an instance of one node is HANG dead.

The log information is as follows:

Fri Feb 28 19:49:04 2014
ARC1: Error 19504 Creating archive log file to '+ data02'
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl1-Archival Error
ORA-16038: log 14 sequence #68244 cannot be archived
ORA-19504: failed to create file ""
ORA-00312: online log 14 thread 1: '+ DATA02/orcl/onlinelog/group_14.264.792274883'
ORA-00312: online log 14 thread 1: '+ DATA02/orcl/onlinelog/group_14.265.792274889'
Archiver process freed from errors. No longer stopped
Fri Feb 28 19:50:22 2014
ARC0: LGWR is actively archiving destination LOG_ARCHIVE_DEST_3
ARCH: Archival stopped, error occurred. Will continue retrying
ORACLE Instance orcl1-Archival Error
ORA-16014: log 14 sequence #68244 not archived, no available destinations
ORA-00312: online log 14 thread 1: '+ DATA02/orcl/onlinelog/group_14.264.792274883'
ORA-00312: online log 14 thread 1: '+ DATA02/orcl/onlinelog/group_14.265.792274889'
ARC0: Archive log rejected (thread 1 sequence 68240) at host 'orclsh'
FAL [server, ARC0]: FAL archive failed, see trace file.
ARCH: FAL archive failed. Archiver continuing
ORACLE Instance orcl1-Archival Error. Archiver continuing.

Analysis:

Because the archive fails to occur on the ASM disk, first check the ASM disk space and DB_RECOVERY_FILE_DEST_SIZE. The ASM disk space is sufficient because there is only one node and the archive fails, it can also be ruled out as a result of insufficient space. Make sure that the DB_RECOVERY_FILE_DEST_SIZE parameter of the two nodes is set to 0. Basically, it can be determined that the problem is related to the abnormal ASM instance status of the current node.

Check the error message of the ASM instance:

Fri Feb 28 19:41:23 2014
Dumping diagnostic data in directory = [cdmp_20130702164115], requested by (instance = 2, osid = 2032294 (LMD0), summary = [incident = 165521].
Fri Feb 28 19:49:19 2014
Dumping diagnostic data in directory = [cdmp_20130702164845], requested by (instance = 2, osid = 2032294 (LMD0), summary = [incident = 165522].
Fri Feb 28 19:55:56 2014
Dumping diagnostic data in directory = [cdmp_20130702165517], requested by (instance = 2, osid = 2032294 (LMD0), summary = [incident = 165523].

This message is displayed on the ASM instance of the current node, indicating that the error occurred on instance 2:

Fri Feb 28 18:34:25 2014
Errors in file/u01/app/grid/diag/asm/+ ASM2/trace/+ ASM2_lmd0_2032294.trc (incident = 186256 ):
ORA-04031: unable to allocate 3768 bytes of shared memory ("shared pool", "unknown object", "sga heap ()", "ges enqueues ")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Insufficient shared pool to allocate a GES object (ospid 2032294)
Fri Feb 28 18:29:53 2014
Sweep [inc] [186256]: completed
Fri Feb 28 18:36:49 2014
Errors in file/u01/app/grid/diag/asm/+ ASM2/trace/+ ASM2_lmd0_2032294.trc (incident = 186257 ):
ORA-04031: unable to allocate 3768 bytes of shared memory ("shared pool", "unknown object", "sga heap ()", "ges enqueues ")
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Insufficient shared pool to allocate a GES object (ospid 2032294)