Oracle Frequent generate a lot of cdmp * directories contain * bucket trace in bdump problem analysis

Recently, a set of database directories were suddenly used with an alarm. It was found that the TRACE Directory was in high usage. It was found that a cdmp * directory is generated every minute in the bdump directory, and each directory contains a large number of files, this is a set of oracle 11.2.0.3 rac on hpux 11.31 databases. If dump event is not enabled in the system, it may be a bug, which is a simple record here.

Oracle @ anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace> ls
Alert_weejar2.log cdmp_20160408221002 cdmp_20160409050508 cdmp_20160427102501 cdmp_20160503133006 cdmp_20160506124003
Cdmp_20160407113747 cdmp_20160408221101 cdmp_20160409050609 cdmp_20160427102602 cdmp_20160503133105 cdmp_20160506124103
Cdmp_20160407111_1 cdmp_20160408221202 cdmp_20160409050708 cdmp_20160427103304 cdmp_20160503133205 cdmp_20160506124202
...

Oracle @ anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502> ls
Weejar2_acms_15550_bucket.trc weejar2_ora_16823_bucket.trc weejar2_ora_180020.bucket.trc
Weejar2_ora_19458_bucket.trc weejar2_ora_26892_bucket.trc weejar2_ora_29042_bucket.trc
Weejar2_acms_15550_bucket.trm weejar2_ora_16823_bucket.trm weejar2_ora_180020.bucket.trm
Weejar2_ora_19458_bucket.trm weejar2_ora_26892_bucket.trm weejar2_ora_29042_bucket.trm
...

Oracle @ anbob2:/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502> cat weejar2_ora_8317_bucket.trc
Trace file/oracle/app/oracle/diag/rdbms/weejar/weejar2/trace/cdmp_20160509173502/weejar2_ora_8317_bucket.trc
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0-64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
And Real Application Testing options
ORACLE_HOME =/oracle/app/oracle/product/11.2.0.3/dbhome_1
System name: HP-UX
Node name: anbob2
Release: B .11.31
Version: U
Machine: ia64
Instance name: weejar2
Redo thread mounted by this instance: 2
Oracle process number: 531
Unix process pid: 8317, image: oracle @ anbob2

* ** 17:35:07. 273
* ** Session id: (832.4201) 17:35:07. 273
 

* ** 17:35:07. 273
Process diagnostic dump for oracle @ anbob2, OS id = 8317,
Pid: 531, proc_ser: 66, sid: 832, sess_ser: 4201
-------------------------------------------------------------------------------
Current SQL:
Client details:
O/S info: user: weblogic, term: unknown, ospid: 1234
Machine: qmweb86 program: JDBC Thin Client
Application name: JDBC Thin Client, hash value = 2546894660
Current Wait Stack:
0: waiting for 'SQL * Net message from client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 55 seq_num = 56 snap_id = 1
Wait times: snap = 4.499210 sec, exc = 4.499210 sec, total = 4.499210 sec
Wait times: max = infinite, heur = 4.499210 sec
Wait counts: cballs = 0 OS = 0
In_wait = 1 iflags = 0x1a0
Wait State:
Fixed_waits = 0 flags = 0x22 boundary = 0x0000000000000000/-1
Session Wait History:
Elapsed time of 0.000031 sec since current wait
0: waited for 'SQL * Net message to client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 54 seq_num = 55 snap_id = 1
Wait times: snap = 0.000001 sec, exc = 0.000001 sec, total = 0.000001 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000012 sec of elapsed time
1: waited for 'SQL * Net message from client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 53 seq_num = 54 snap_id = 1
Wait times: snap = 0.001062 sec, exc = 0.001062 sec, total = 0.001062 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000027 sec of elapsed time
2: waited for 'SQL * Net message to client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 52 seq_num = 53 snap_id = 1
Wait times: snap = 0.000001 sec, exc = 0.000001 sec, total = 0.000001 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000016 sec of elapsed time
3: waited for 'SQL * Net message from client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 51 seq_num = 52 snap_id = 1
Wait times: snap = 0.001342 sec, exc = 0.001342 sec, total = 0.001342 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000526 sec of elapsed time
4: waited for 'SQL * Net message to client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 50 seq_num = 51 snap_id = 1
Wait times: snap = 0.000001 sec, exc = 0.000001 sec, total = 0.000001 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000173 sec of elapsed time
5: waited for 'SQL * Net message from client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 49 seq_num = 50 snap_id = 1
Wait times: snap = 0.000847 sec, exc = 0.000847 sec, total = 0.000847 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000003 sec of elapsed time
6: waited for 'SQL * Net message to client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 48 seq_num = 49 snap_id = 1
Wait times: snap = 0.000002 sec, exc = 0.000002 sec, total = 0.000002 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000041 sec of elapsed time
7: waited for 'SQL * Net message from client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 47 seq_num = 48 snap_id = 1
Wait times: snap = 0.029814 sec, exc = 0.029814 sec, total = 0.029814 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000076 sec of elapsed time
8: waited for 'SQL * Net message to client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 46 seq_num = 47 snap_id = 1
Wait times: snap = 0.000001 sec, exc = 0.000001 sec, total = 0.000001 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000153 sec of elapsed time
9: waited for 'SQL * Net message from client'
Driver id = 0x74637000, # bytes = 0x1, = 0x0
Wait_id = 45 seq_num = 46 snap_id = 1
Wait times: snap = 0.002608 sec, exc = 0.002608 sec, total = 0.002608 sec
Wait times: max = infinite
Wait counts: cballs = 0 OS = 0
Occurred after 0.000002 sec of elapsed time
Sampled Session History of session 832 serial 4201
---------------------------------------------------
The sampled session history is constructed by sampling
The target session every 1 second. The sampling process
Captures at each sample if the session is in a non-idle wait,
An idle wait, or not in a wait. If the session is in
Non-idle wait then one interval is shown for all the samples
The session was in the same non-idle wait. If
Session is in an idle wait or not in a wait
Consecutive samples then one interval is shown for all
The consecutive samples. Though we display these consecutive
Samples in a single interval the session may NOT be continuously
Idle or not in a wait (the sampling process does not know ).
 
The history is displayed in reverse chronological order.
 
Sample interval: 1 sec, max history 120 sec
---------------------------------------------------
[121 samples, 17:33:07-17:35:07]
Idle wait at each sample
-------------------------------------------------------------------------------
Process diagnostic dump actual duration = 0.005000 sec
(Max dump time = 30.000000 sec)

* ** 17:35:07. 278
 
-------------------------------------------------------------------------------
Trace Bucket Dump Begin: default bucket for process 531 (osid: 8317)
TIME (* = approx): SEQ: COMPONENT: FILE @ LINE: FUNCTION: SECT/DUMP: [EVENT #: PID: SID] DATA
-------------------------------------------------------------------------------
17:32:17. 785270: GSIPC: kjctc. c @ 321: kjctcr (): GSIPC: CONN: Disconnect from inst 1, receiver 2
17:32:17. 785271: 89F5D5D1: db_trace: ksxp. c @ 4368: ksxpmsgcncl (): [10401: 531: 0] KSXPMSGCNCL: client 2 tid inst 1 ptid 257 flags 0x100
17:32:17. 785271: GSIPC: kjctc. c @ 321: kjctcr (): GSIPC: CONN: Disconnect from inst 1, receiver 3
17:32:17. 785272: 89F5D5D2: db_trace: ksxp. c @ 4368: ksxpmsgcncl (): [10401: 531: 0] KSXPMSGCNCL: client 2 tid inst 1 ptid 257 flags 0x100
17:32:17. 785569: GSIPC: kjcc. c @ 626: kjccdel (): GSIPC: CLNT: deleted client comm layer structures 2
17:32:17. 785576: 89F5D5D4: db_trace: kss. c @ 1414: kssdch (): [10809: 531: 0] kssdch (0xc0000010e1d0bdc8 = process, 2) 2 0 exit

View system event dump

SQL> oradebug setmypid
Statement processed.
SQL> oradebug eventdump system
10949 trace name context forever, level 1
28401 trace name context forever, level 1
SQL> oradebug eventdump process
28401 trace name context forever, level 1
10949 trace name context forever, level 1
SQL> oradebug eventdump session
28401 trace name context forever, level 1
10949 trace name context forever, level 1
A bug was found in Mos. Bug 17062524 Diagnostic enhancement to write default trace bucket to trace file for database

The default RDBMS tracing bucket is not dumped to the trace file on
Failure-instead it is dumped to a cdmp * file.

Workaround
Find the cdmp * file containing the default tracing bucket's contents. (This
File is anot encoded Ded by default when packaging an incident so has to be
Collected manually)

Fixed:
12.2 (Future Release)
12.1.0.2 (Server Patch Set)