Ora-00600 [KGHLKREM1]

Ora-00600 [KGHLKREM1]

Symptom:
Alert Log error:
Sat May 18 07:59:47 2013
Errors in file/oracle/PP/saptrace/usertrace/pp_ora_rj630.trc:
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
Sat May 18 07:59:51 2013
Errors in file/oracle/PP/saptrace/usertrace/PP_ora_4530210.trc:
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
ORA-01017: invalid username/password; logon denied
Sat May 18 07:59:52 2013
Errors in file/oracle/PP/saptrace/usertrace/PP_ora_4571226.trc:
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
Sat May 18 07:59:55 2013
Errors in file/oracle/PP/saptrace/usertrace/PP_ora_3580036.trc:
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
Sat May 18 07:59:55 2013
Errors in file/oracle/PP/saptrace/background/PP_smon_3174646.trc:
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
Sat May 18 07:59:55 2013
Errors in file/oracle/PP/saptrace/usertrace/pp_ora_0000634.trc:
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
ORA-01017: invalid username/password; logon denied
Sat May 18 07:59:59 2013
Errors in file/oracle/PP/saptrace/background/PP_pmon_3006490.trc:
ORA-00474: SMON process terminated with error.
Sat May 18 07:59:59 2013
PMON: terminating instance due to error 474
Instance terminated by PMON, pid = 3006490

PP_ora_3477608.trc information:
Oracle Database 10g Enterprise Edition Release 10.2.0.2.0-64bit Production
With the Partitioning and Data Mining options
ORACLE_HOME =/oracle/PP/102_64
System name: AIX
Node name: ppserver01
Release: 3
Version: 5
Machine: 00C2E56F4C00
Instance name: PP
Redo thread mounted by this instance: 1
Oracle process number: 38
Unix process pid: 3477608, image: oraclePP @ ppserver01

* ** 07:49:04. 855
* ** Action name: (298) 07:49:04. 854
* ** Module name :( GP4IV83BY2WCKQIJN2P2XT31F5T) 07:49:04. 854
* ** Service name :( SYS $ USERS) 07:49:04. 854
* ** Session id: (1252.62448) 07:49:04. 854
***** Internal heap ERROR KGHLKREM1 addr = 7000008c2000020 ds = 700000010000058 *****
* *** Dump of memory around addr 7000008c2000020:

7000008C2000000 00000000 00000000 00000000 00000000 [......]
Repeat 1 times
7000008C2000020 07000000 10001990 00000000 00000000 [......]
7000008C2000030 00000000 00000000 00000000 00000000 [......]

**************************************** **************
Heap dump heap name = "sga heap" desc = 700000010000058
Extent sz = 0xfe0 alt = 216 het = 32767 rec = 9 flg =-126 opc = 4
Parent = 0 owner = 0 nex = 0 xsz = 0x20
* ** 07:49:04. 866
Ksedmp: internal or fatal error
ORA-00600: internal error code, arguments: [KGHLKREM1], [0x7000008C2000020], [], [], [], [], [], [], []
Current SQL statement for this session:
SELECT/* + FIRST_ROWS (1) */"COMP_CODE", "CHRT_ACCTS", "GL_ACCOUNT", "FISCVARNT", "FISCPER", "AC_DOC_NO", "ITEM_NUM ", "FI_DOCSTAT", "ACCT_TYPE", "SP_GL_IND", "AC_DOC_TYP", "POST_KEY", "DOC_DATE", "PSTNG_DATE", "CREATEDON", "CLEAR_DATE", "CLR_DOC_NO ", "CO_AREA", "COSTCENTER", "PROFIT_CTR", "COORDER", "WBS_ELEMT", "PLANT", "BUS_AREA", "FUNC_AREA", "PART_PRCTR", "PCOMPANY ", "PBUS_AREA", "LOC_CURRCY", "DEBIT_LC", "CREDIT_LC", "DEB_CRE_LC", "region", "DEBIT_DC", "CREDIT_DC", "DEB_CRE_DC", "region ", "comment", "comment", "CREDIT_LC2", "DEB_CRE_L2", "LOC_CURTP3", "LOC_CURRC3", "comment", "CREDIT_LC3", "DEB_CRE_L3", "REF_DOC_NO ", "REF_KEY3", "ORG_DOC_NO", "example", "ORG_DOC_CC", "ORG_DOC_CO", "RECORDMODE", "ALLOC_NMBR", "POSTXT", "USERNAME", "DOC_HD_TXT ", "ASSET_MAIN", "ASSET", "REASON_CDE", "/BIC/ZTCODE", "FI_DBCRIND ", "/BIC/ZAWKEY" FROM "/BIC/AFI00_O0000" WHERE "COMP_CODE" =: A0 AND "CHRT_ACCTS" =: A1 AND "GL_ACCOUNT" =: A2 AND "FISCVARNT" =: a3 AND "FISCPER" =: A4 AND "AC_DOC_NO" =: A5 AND "ITEM_NUM" =: A6
----- Call Stack Trace -----
Calling call entry argument values in hex
Location type point (? Means dubious value)
----------------------------------------------------------------------------

Kghnerror + 0470 bl kgesin 11016A6C8? 1010000e2ac? <=== Call stack: kghnerror kghadd_reserved_extent kghget_reserved_extent kghgex kghfnd kghalo kghgex kghalf kghphosphatase kglsal kglsadd
000000000? 100000001?
000000002? 7000008C2000020?
000000000? 000000000?
Kghadd_reserved_ext bl kghnerror 000000000? 700000010003B98?
Ent + 0244 1100012A8? 7000000100F91F8?
Kghget_reserved_ext bl kghadd_reserved_ext 000000000? 7000009C975E740?
Ent + 0214 ent 00000001B? 000000000?
100000001b? 000000008?
Kghgex + 0534 bl kghget_reserved_ext ffffffffea3d0? 1101F0240?
Ent 1102273A0? 700000997C995F0?
000008000?
Kghfnd + 05ac bl kghgex ffffffffea4c0? 0000008E8?
000000018? 000000001?
000000001? 000000000?
09007 AFFF?
Kghalo + 0a14 bl kghfnd 000000000? 000000000?
000000000? 000000000?
000000000? 000000000?
Kghgex + 036c bl kghaluo 000000002? 000000000?
7000009C975E740? 00000000B?
0000000C7? FFFFFFFFFEAEB8?
Ffffffffeac98? 00C87A838?
Kghalf + 0460 bl kghgex 000008000? 1043EC0FC?
1043EC0F4? 70000080C9B02E0?
000000000? 000000090?
70000080C9B02E8?
Kghphosphatase + 0278 bl kghalf 000000000? 70000080C87A838?
000000000? 000000005?
000aa? FFFFFFFFFFEB080?
Kglsal + 00a4 bl kghphosphatase 70000080C9B02E8? 000000004?
000000000? 70000080C87A838?
70000080C87A906? 000000001?
Kglsadd + 00ec bl kglsal 70000080C87A906? 000000001?

----------------------------------------
SO: 7000009ca7702d0, type: 4, owner: 7000009c851ec40, flag: INIT/-/0x00
(Session) sid: 1494 trans: 0, creator: 7000009c851ec40, flag: (1000041) USR /--/-/-/-/-/-
DID: 0001-0026-00058CCA, short-term DID: 0000-0000-00000000
Txn branch: 0
Oct: 0, prv: 0, SQL: 0, psql: 700000946c2ba88, user: 27/SAPSR3
O/S info: user: PPadm, term:, ospid: 4649210, machine: ppserver01
Program: dw. sapPP_DVEBMGS01 @ ppserver01 (TNS V1-V3)
Client info: 1
Application name: GP4F1RTDKREPXII6BJ25XA6U9K6, hash value = 2112637063
Action name: 82 and hash value = 2257323982
Last wait for 'SQL * Net message from client' blocking sess = 0x0 seq = 45555 wait_time = 142223 seconds since wait started = 116
Driver id = 54435000, # bytes = 1, = 0
Dumping Session Wait History
For 'SQL * Net message from client' count = 1 wait_time = 142223
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message to client' count = 1 wait_time = 1
Driver id = 54435000, # bytes = 1, = 0
For 'Log file sync' count = 1 wait_time = 26280
Buffer # = 2e98, = 0, = 0
For 'SQL * Net message from client' count = 1 wait_time = 191
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message to client' count = 1 wait_time = 1
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message from client' count = 1 wait_time = 267
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message to client' count = 1 wait_time = 1
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message from client' count = 1 wait_time = 242
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message to client' count = 1 wait_time = 1
Driver id = 54435000, # bytes = 1, = 0
For 'SQL * Net message from client' count = 1 wait_time = 361
Driver id = 54435000, # bytes = 1, = 0
Temporary object counter: 0
----------------------------------------

* ****************** Session Cursor Dump ***************** *****
Current cursor: 198, pgadep: 0
Open cursors (pls, sys, hwm, max): 236 (0, 3,256,200 0)
NULL 9 SYNTAX 1 PARSE 0 BOUND 223 FETCH 0 ROW 3
Cached frame pages (total, free ):
4 k (20, 18), 8 k (1, 1), 16 k (1, 1), 32 k (0, 0)
Pgactx: 70000095492a1d8 ctxcbk: 0 ctxqbc: 11050ab98 ctxrws: 0

----------------------------------------
Cursor #198 (110613388) state = SYNTAX curiob = 11062f3f8
Curflg = 4e fl2 = 0 par = 0 ses = 7000009c971af60
Child cursor: 170
Sqltxt (7000008e336c8a0) =
SELECT/* + FIRST_ROWS (1) */"COMP_CODE", "CHRT_ACCTS", "GL_ACCOUNT", "FISCVARNT", "FISCPER", "AC_DOC_NO", "ITEM_NUM ", "FI_DOCSTAT", "ACCT_TYPE", "SP_GL_IND", "AC_DOC_TYP", "POST_KEY", "DOC_DATE", "PSTNG_DATE", "CREATEDON", "CLEAR_DATE", "CLR_DOC_NO ", "CO_AREA", "COSTCENTER", "PROFIT_CTR", "COORDER", "WBS_ELEMT", "PLANT", "BUS_AREA", "FUNC_AREA", "PART_PRCTR", "PCOMPANY ", "PBUS_AREA", "LOC_CURRCY", "DEBIT_LC", "CREDIT_LC", "DEB_CRE_LC", "region", "DEBIT_DC", "CREDIT_DC", "DEB_CRE_DC", "region ", "comment", "comment", "CREDIT_LC2", "DEB_CRE_L2", "LOC_CURTP3", "LOC_CURRC3", "comment", "CREDIT_LC3", "DEB_CRE_L3", "REF_DOC_NO ", "REF_KEY3", "ORG_DOC_NO", "example", "ORG_DOC_CC", "ORG_DOC_CO", "RECORDMODE", "ALLOC_NMBR", "POSTXT", "USERNAME", "DOC_HD_TXT ", "ASSET_MAIN", "ASSET", "REASON_CDE", "/BIC/ZTCODE", "FI_DBCRIND ", "/BIC/ZAWKEY" FROM "/BIC/AFI00_O0000" WHERE "COMP_CODE" =: A0 AND "CHRT_ACCTS" =: A1 AND "GL_ACCOUNT" =: A2 AND "FISCVARNT" =: a3 AND "FISCPER" =: A4 AND "AC_DOC_NO" =: A5 AND "ITEM_NUM" =: A6
Hash = e827ac0da76d2a7a9df64f5504a0c7dc
Parent = 7000009b5e24428 maxchild = 03 plk = 7000009b896ab70 ppn = n
Cursor instantiation = 11062f3f8 used = 1368834544
Child #0 (7000008a97c3438) pcs = 7000009b5e24038
Clk = 7000009b878e490 ci = 7000009b5e23720 pn = 700000946425768 ctx = 70000095492a1d8
Kgsccflg = 0 llk [11062f400, 11062f400] idx = 0
Xscflg = 130437 fl2 = 41240009 fl3 = 2022000 fl4 = 80
Bind bytecodes
Kkscoacd
Bind #0
Oacdty = 96 mxl = 32 (12) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 224 off = 0
No bind buffers allocated
Bind #1
Oacdty = 96 mxl = 32 (12) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 0 off = 32
No bind buffers allocated
Bind #2
Oacdty = 96 mxl = 32 (30) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 0 off = 64
No bind buffers allocated
Bind #3
Oacdty = 96 mxl = 32 (06) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 0 off = 96
No bind buffers allocated
Bind #4
Oacdty = 96 mxl = 32 (21) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 0 off = 128
No bind buffers allocated
Bind #5
Oacdty = 96 mxl = 32 (30) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 0 off = 160
No bind buffers allocated
Bind #6
Oacdty = 96 mxl = 32 (09) mxlc = 00 mal = 00 scl = 00 pre = 00
Oacflg = 03 fl2 = 1000010 frm = 01 csi = 871 siz = 0 off = 192
No bind buffers allocated
Frames pfr 110547e90 siz = 10056 efr 110547dc0 siz = 10032
Cursor frame dump
Enxt: 5.0x00000010
Pnxt: 2.0x00000008 pnxt: 1.0x00000010
Kxscphp 1106b4dc8 siz = 6552 inu = 6000 nps = 5856
Kxscdfhp 1105b5be0 siz= 984 inu = 88 nps = 0

Analysis:
A known BUG in Oracle:
Bug 6137313: smon crashed with ORA-600 [KGHLKREMF () + 33] AFTER ORA-600 [KGHLKREM1]
BUG: 5736850 multi kill session leads to sga resume uption

Solution provided by Oracle:
1. Upgrade to a supported release, 11.2.0.3
Or
2. Apply the 10.2.0.5.0 patchset
Or
3. Use the known workaround of setting:
SQL> alter system set "_ bloom_filter_enabled" = false scope = both;