SO:0X5533EFEA0, Type:4, owner:0x554631060, flag:init/-/-/0x00 if:0x3 c:0x3
proc=0x554631060, Name=session, File=ksu.h line:12624 ID:, pg=0
(session) sid:388 Ser:1 trans:0x0, creator:0x554631060
Flags: (0x51) usr/-flags_idl: (0x1) bsy/-/-/-/-/-
FLAGS2: (0x40409)-/-/inc
Did:, short-term did:
Txn branch:0x0
Oct:3, prv:0, sql:0x5369ece68, psql:0x450616048, User:0/sys
Ksuxds FALSE at location:0
Service Name:sys$background
Current Wait Stack:
0:waiting for ' Enq:pv-syncstart '
name|mode=0x50560006, 0=0x0, 0=0x0
wait_id=8642197 seq_num=64811 snap_id=1
Wait times:snap=3934 min to sec, exc=3934 min. sec, total=3934 min sec
Wait Times:max=infinite, heur=3934 min. sec
Wait counts:calls=1088 os=1088
In_wait=1 iflags=0x15a0
There is at the least one session blocking this session.
Dumping 1 direct blocker (s):
Inst:1, sid:773, ser:1
Dumping final blocker:
Inst:1, sid:397, Ser:7
----------------------------------------
So:0x553857f80, Type:4, Owner:0x55463e8e8, flag:init/-/-/0x00 if:0x3 c:0x3
Proc=0x55463e8e8, Name=session, File=ksu.h line:12624 ID:, pg=0
(session) sid:773 Ser:1 trans:0x0, Creator:0x55463e8e8
Flags: (0x51) usr/-flags_idl: (0x1) bsy/-/-/-/-/-
FLAGS2: (0x409)-/-/inc
Did:, short-term did:
Txn branch:0x0
oct:0, prv:0, sql:0x0, psql:0x0, User:0/sys
Ksuxds FALSE at location:0
Service Name:sys$background
Current Wait Stack:
0:waiting for ' enq:pr-contention '
name|mode=0x50520006, 0=0x0, 0=0x0
wait_id=410416 seq_num=934 snap_id=1
Wait times:snap=3934 min exc=3934 sec, total=3934 min (sec)
Wait Times:max=infinite, heur=3934 min (sec)
Wait counts:calls=76 os=76
In_wait=1 iflags=0x15a0
There is at the least one session blocking this session.
Dumping 1 direct blocker (s):
Inst:1, sid:397, Ser:7
Dumping final blocker:
Inst:1, sid:397, Ser:7
There is 1 sessions blocked by this session.
Dumping One waiter:
Inst:1, sid:388, ser:1
Wait event: ' Enq:pv-syncstart '
P1: ' Name|mode ' =0x50560006
P2: ' 0 ' =0x0
P3: ' 0 ' =0x0
PROCESS 45:cjq0
----------------------------------------
So:0x554645d80, Type:2, owner:0x0, flag:init/-/-/0x00 if:0x3 c:0x3
Proc=0x554645d80, name=process, File=ksu.h line:12616 ID:, pg=0
(process) Oracle pid:45, Ser:2, calls Cur/top:0x5394a48b8/0x5394a48b8
Flags: (0x2) SYSTEM
FLAGS2: (0x0), FLAGS3: (0x10)
Intr error:0, call error:0, Sess error:0, txn error 0
Intr Queue:empty
KSUDLP FALSE at location:0
(post info) Last Post received:0 0 9
Last Post Received-location:ksq.h line:2014 ID:KSQRCL
Last process to post me:554631060 1 2
Last Post sent:0 0 9
Last Post Sent-location:ksq.h line:2014 ID:KSQRCL
Last process posted by me:554631060 1 2
(latch info) wait_event=0 bits=0
Process Group:default, Pseudo Proc:0x554a379f8
O/S info:user:oracle, Term:unknown, ospid:8914
OSD PID Info:unix Process pid:8914, Image: [email protected] (CJQ0)
SO:0X5533D4FC0, Type:4, Owner:0x554645d80, flag:init/-/-/0x00 if:0x3 c:0x3
Proc=0x554645d80, Name=session, File=ksu.h line:12624 ID:, pg=0
(session) sid:397 Ser:7 trans:0x0, Creator:0x554645d80
Flags: (0x51) usr/-flags_idl: (0x1) bsy/-/-/-/-/-
FLAGS2: (0x40409)-/-/inc
Did:, short-term did:
Txn branch:0x0
oct:0, prv:0, sql:0x0, Psql:0x450caef60, User:0/sys
Ksuxds FALSE at location:0
Service Name:sys$background
Current Wait Stack:
0:waiting for ' OS thread startup '
=0x0, =0x0, =0x0
wait_id=2782824 seq_num=30355 snap_id=1
Wait times:snap=3934 min sec, exc=3934 min sec, total=3934 min sec
Wait times:max=2 min 0 sec, heur=3934 min sec
Wait counts:calls=77 os=77
In_wait=1 iflags=0x15a0
There is 4 sessions blocked by this session.
Dumping One waiter:
Inst:1, sid:773, ser:1
Wait event: ' Enq:pr-contention '
P1: ' Name|mode ' =0x50520006
P2: ' 0 ' =0x0
P3: ' 0 ' =0x0
row_wait_obj#: 4294967295, block#: 0, row#: 0, file# 0
min_blocked_time:0 secs, waiter_cache_ver:63462
Wait State:
Fixed_waits=0 flags=0x22 boundary=0x0/-1
Session Wait History:
Elapsed time of 0.000160 sec since current wait
0:waited for ' enq:pr-contention '
name|mode=0x50520006, 0=0x0, 0=0x0
wait_id=2782823 seq_num=30354 snap_id=1
Wait times:snap=4 min. sec, Exc=4 min. sec, Total=4 min. sec
Wait Times:max=infinite
Wait counts:calls=8 os=8
Occurred after 2.476248 sec of elapsed time
1:waited for ' OS thread startup '
=0x0, =0x0, =0x0
wait_id=2782822 seq_num=30353 snap_id=1
Wait times:snap=41 min (sec), Exc=41 min (sec), total=41 min sec
Wait times:max=2 min 0 sec
Wait counts:calls=2 os=2
Occurred after 0.000168 sec of elapsed time
2:waited for ' enq:pr-contention '
Dumping One waiter:
Inst:1, sid:773, ser:1
Wait event: ' Enq:pr-contention '
P1: ' Name|mode ' =0x50520006
P2: ' 0 ' =0x0
P3: ' 0 ' =0x0
row_wait_obj#: 4294967295, block#: 0, row#: 0, file# 0
min_blocked_time:0 secs, waiter_cache_ver:63462
Database internal wait Relationship:
blocking Blocker Event
388 773,397 Enq:pv-syncstart
773 397 Enq:pr-contention
397 enq:pr-contention OS Thread startup:ora-445
--1
Node 1 run statement select sid,serial# from gv$session where username= ' Wanqiong ':
Reason for failure: Process 53/572 sessions run the statement, Process 20:CKPT block these two sessions
PROCESS 20 due to waiting for ' enq:pr-contention ' name|mode=0x50520006
0x50520006 owner: PROCESS 45:cjq0
According to: awk-f Ass109.awk CALLDB1_ORA_15639.TRC Analysis:
Resource Holder State
Enqueue pr-00000000-00000000 45:0: Waiting for ' OS thread startup '
Enqueue pv-00000000-00000000 38:38:is waiting for 45:
Latch 380030bc0??? Blocker
Process 45:cjq0 progress is the culprit! The process could not be started.
--2 Node 2
Node 2 run statement: Select SUM (bytes) from dba_segments, wait time 0:waiting for ' GC CR request '
CSSD,CRSD not found obvious node communication error, suspected due to Node 1 crash caused
--3 Node 1 warning log analysis
Mon 11 09:45:49 2014
Archived Log Entry 58557 added for thread 1 sequence 27907 ID 0x4bcd9425 dest 1:
Mon 11 16:08:44 2014
Errors in File/oracle/diag/rdbms/calldb/calldb1/trace/calldb1_cjq0_8914.trc (incident=56362):
Ora-00445:background process "J000" did not start after the seconds
Incident details in:/oracle/diag/rdbms/calldb/calldb1/incident/incdir_56362/calldb1_cjq0_8914_i56362.trc
Kkjcre1p:unable to spawn JOBQ slave process
Errors in FILE/ORACLE/DIAG/RDBMS/CALLDB/CALLDB1/TRACE/CALLDB1_CJQ0_8914.TRC:
Mon 11 16:12:25 2014
Errors in File/oracle/diag/rdbms/calldb/calldb1/trace/calldb1_mmon_8497.trc (incident=56202):
Ora-00445:background process "m000" did not start after the seconds
Incident details in:/oracle/diag/rdbms/calldb/calldb1/incident/incdir_56202/calldb1_mmon_8497_i56202.trc
Mon 11 16:14:28 2014
AWR Analysis Memory 16G
Begin End
Host Mem (MB): 15,744.015,744.0
SGA Use (MB): 4,608.04,608.0
PGA Use (MB): 462.5460.1
% Host Mem used for SGA+PGA: 32.2132.19
Memory 12g after reboot
[Email protected]/home/oracle]$ prtconf | grep ' Memory '
Memory size:12288 Megabytes
Suspect that the problem is in memory:
1379200.1 Depending on the documentation you can determine:
What does this message mean?
The message indicates that we failed to spawn a new process at the Operating System level to serve the request.
There is various causes for this issue. This typically occurs when there are a shortage or misconfiguration in
Operating System Resources, and thereby the problem should is investigated from an OS perspective. However
There is a few causes related to the Oracle Database as well.
The default seconds (after which Oracle times off) can be extended dynamically (without a database restart) b
Setting the following event:
OS Configuration Checks "Yesterday determined hardware is defective"
Checks on Oracle Database Configuration
--Dump Systemstat made before database reboot 10
/oracle/diag/rdbms/calldb/calldb1/trace/calldb1_ora_15639.trc
