ORA-32701 troubleshooting

Check found that a set of rac two instances Using ASM basically reported a ORA-32701 error every hour, the error message in the capture alert Log is as follows:

Check found that a set of rac two instances Using ASM basically reported a ORA-32701 error every hour, the error message in the capture alert Log is as follows:

I. Version Information

Operating system version: AIX 61009

Oracle Database Version: 11.2.0.3.11 (RAC)

Ii. Error description

Check found that a set of rac two instances Using ASM basically reported a ORA-32701 error every hour, the error message in the capture alert Log is as follows:

Sat Dec 06 09:44:00 2014
Errors in file/oracle/app/oracle/diag/rdbms/egmmdb/egmmdb2/trace/egmmdb2_dia0_13500888.trc (incident = 1041128 ):
ORA-32701: Possible hangs up to hang ID = 0 detected
Incident details in:/oracle/app/oracle/diag/rdbms/egmmdb/egmmdb2/incident/incdir_20171128/egmmdb2_dia0_13500888_i1_1128.trc
DIA0 terminating blocker (ospid: 15335610 sid: 1299 ser #: 5849) of hang with ID = 3
Requested by master DIA0 process on instance 1
Hang Resolution Reason: Although the number of affected sessions did not
Justify automatic hang resolution initially, this previusly ignored
Hang was automatically resolved.
By terminating session sid: 1299 ospid: 15335610
Sat Dec 06 09:44:01 2014
Sweep [inc] [1041128]: completed
Sweep [inc2] [1041128]: completed
DIA0 successfully terminated session sid: 1299 ospid: 15335610 with status 31.
Sat Dec 06 09:45:35 2014
Errors in file/oracle/app/oracle/diag/rdbms/egmmdb/egmmdb2/trace/egmmdb2_dia0_13500888.trc (incident = 1041129 ):
ORA-32701: Possible hangs up to hang ID = 0 detected
Incident details in:/oracle/app/oracle/diag/rdbms/egmmdb/egmmdb2/incident/incdir_20171129/egmmdb2_dia0_13500888_i1_1129.trc
DIA0 terminating blocker (ospid: 15335610 sid: 1299 ser #: 5849) of hang with ID = 3
Requested by master DIA0 process on instance 1
Hang Resolution Reason: Although the number of affected sessions did not
Justify automatic hang resolution initially, this previusly ignored
Hang was automatically resolved.
By terminating the process
DIA0 successfully terminated process ospid: 15335610.
Sat Dec 06 09:45:37 2014
Sweep [inc] [1041129]: completed
Sweep [inc2] [1041129]: completed
Sat Dec 06 10:45:12 2014
Errors in file/oracle/app/oracle/diag/rdbms/egmmdb/egmmdb2/trace/egmmdb2_dia0_13500888.trc (incident = 1041130 ):
ORA-32701: Possible hangs up to hang ID = 0 detected
Incident details in:/oracle/app/oracle/diag/rdbms/egmmdb/egmmdb2/incident/incdir_1_1130/egmmdb2_dia0_13500888_i1_1130.trc
Sat Dec 06 10:45:13 2014
Sweep [inc] [1041130]: completed
Sweep [inc2] [1041130]: completed

Extract the following information from egmmdb2_dia0_13500888_i1_1129.trc:

* ** 09:45:35. 770
Resolvable Hangs in the System
Root Chain Total Hang
Hang Inst Root # hung Hang Resolution
ID Type Status Num Sess Conf Span Action
-------------------------------------------------------------------
3 hang rslnpend 2 1299 2 2 high global Terminate Process
Hang Resolution Reason: Although the number of affected sessions did not
Justify automatic hang resolution initially, this previusly ignored
Hang was automatically resolved.

Inst # SessId Ser # OSPID PrcNm Event
-----------------------------------
1 1444 7855 10420452 M000 enq: FU-contention
2 1299 5849 15335610 M000 not in wait <<=== from here, we can see that one M00 * process blocks another M00 * process (from the alert Log above, we can see, hang Manager solves the Hang problem by killing the blocked 1299 sessions)

Dumping process info of pid [155.15335610] (sid: 1299, ser #: 5849)
Requested by master DIA0 process on instance 1.

* ** 09:45:35. 770
Process diagnostic dump for oracle @ egmmdb2 (M000), OS id = 15335610,
Pid: 155, proc_ser: 153, sid: 1299, sess_ser: 5849
-------------------------------------------------------------------------------
OS thread scheduling delay history: (sampling every 1.000000 secs)
0.000000 secs at [09:45:35]
NOTE: scheduling delay has not been sampled for 0.376554 secs 0.000000 secs from [09:45:31-09:45:36], 5 sec avg
0.000000 secs from [09:44:36-09:45:36], 1 min avg
0.000000 secs from [09:40:36-09:45:36], 5 min avg
Loadavg: 2.68 2.42 2.41
Swap info: free_mem = 19881.13 m clr = 256.00 M
Alloc = 138.07 M avail = 65536.00 M swap_free = 65397.93 M
F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD
240001 A oracle 15335610 1 0 60 20 948d16590 209136 f1000a01500d48b0 08:37:22-ora_m000_egmmdb2
Short stack dump:
Ksedsts () + 360 <-ksdxfstk () + 44 <-ksdxcb () + 3384 <-sspuser () + 116 <-49f0 <-sntpread () + 40 <-ntpfprd () + 116 <-nsbasic_brc () + 1116 <-nsbrecv () + 128 <-nioqrc () + 17480 <-ttcdrv () + 988 <-nioqwa () + 76 <-upirtrc () + 2428 <-kpurcsc () + 108 <-kpuexec () + 3088 <-OCIStmtExecute () + 32 <-kfgGrpTableCbInternal () + 2756 <-kfgGrpTableCb () + 48 <-qerfxFetch () + 3728 <-qergsFetch () + 6856 <-opifch2 () + 6228 <-opiefn0 () + 552 <-opipls () + 3492 <-opiodr () + 720 <-rpidrus () + 216 <-skgmstack () + 120 <-rpidru () + 132 <-rpiswu2 () + 480 <-rpidrv () + 1060 <-psddr0 () + 260 <-psdnal () + 472 <-pevm_EXECC () + 532 <-pfr1__execc () + 72 <-pfrrun_no_tool () + 96 <-pfrrun () + 1032 <-plsql_run () + 616 <-peicnt () + 320 <-kkxexe () + 508 <-opiexe () + 17224 <-opiall0 () + 1296 <-opikpr () + 488 <-opiodr () + 720 <-rpidrus () + 216 <-skgmstack () + 120 <-rpidru () + 132 <-rpiswu2 () + 480 <-kprball () + 1036 <-kkxsexr () + 380 <-kkxsexe () + 604 <-pevm_icd_call_common () + 896 <-pfrinstr_ICAL () + 156 <-pfrrun_no_tool () + 96 <-pfrrun () + 1032 <-plsql_run () + 616 <-peicnt () + 320 <-kkxexe () + 508 <-opiexe () + 17224 <-kpoal8 () + 4616 <-opiodr () + 720 <-kpoodrc () + 36 <-rpiswu2 () + 480 <-kpoodr () + 512 <-upirtrc () + 1328 <-kpurcsc () + 108 <-kpuexec () + 10448 <-OCIStmtExecute () + 32 <-kewuxs_execute_statement () + 676 <-example () + 268 <-kewfads_auto_dbfus_slave () + 1296 <-kebm_slave_main () + 492 <-ksvrdp () + 1708 <-opirip () + 724 <-opidrv () + 608 <-sou2o () + 136 <-opimai_real () + 188 <-ssthrdmain () + 268 <-main () + 204 <-_ start () + 112

-------------------------------------------------------------------------------
Process diagnostic dump actual duration = 0.084000 sec
(Max dump time = 15.000000 sec)