Analysis Case: Application Server w3wp process CPU lasts more than 90% (Oracle client Bug)

Problem Description:

Project feedback The response of one of the server business operations to the application payload is very slow, and logging on to the server finds that the W3WP process CPU continues to exceed 90%, even during low peak business periods. Remote view found that the application server hosting the request is really low, why???

Cause Analysis:

The dump of the crawl w3wp process found that the running thread did not have the stack code of our system. And the top of the stack of long-running worker threads is basically Oracle.DataAccess.Client.OracleTuningAgent.DoScan ()----" Oracle.DataAccess.Client.OracleTuningAgent.TuningFunction () related calls.

According to Oracle.DataAccess.Client.OracleTuningAgent.TuningFunction () and other code prompts to consult the data, it appears to be related to the Oracle server and client version, after comparing the application and other application server, The Oracle client version on the application server that was found to be problematic is indeed low (11.2.0.1), and the other Oracle client version that responds to the normal application server is 11.2.0.3. The problem disappears after contacting the site to upgrade the Oracle client version.

Reference: http://stackoverflow.com/questions/2782169/oracle-data-provider-pegs-iis-worker-process-when-web-site-is-stopped

This have been fixed in 11.2.0.2 and in Patch 9966926 ORACLE 11G 11.2.0.1 patch 5 bugs for WINDOWS (64-bit AMD64 and INTEL E M64T).

Or Workaround:is to disable-tuning by adding ' self tuning=false ' to the connection string.

The dump log is as follows:

0:000>!runaway User Mode time  Thread   time

60:13FDC 2 days 19:54:05.92262:3314 2 days 19:25:29.44364:e0c 2 days 16:46:44.319

  75:EF2C       0 days 2:06:17.155  67:198ac      0 days 1:54:59.175  76:1afe4      0 days 1:41:53.460  20:19f0c      0 days 0:04:41.1450x13fdc

Child SP IP call SITE0000000030C9DCC8 00000000779cd3fa[INLINEDCALLFRAME:0000000030C9DCC8]System.Exception.GetMessageFromNativeResources (Exceptionmessagekind,System.Runtime.CompilerServices.StringHandleOnStack) 0000000030C9DCC8 000007fef86e5aca[INLINEDCALLFRAME:0000000030C9DCC8]System.Exception.GetMessageFromNativeResources (Exceptionmessagekind,System.Runtime.CompilerServices.StringHandleOnStack) 0000000030C9DCA0 000007fef86e5aca * * * * warning:unable to Verify checksum for mscorlib.ni.dllSystem.Threading.ThreadAbortException. ctor () 0000000030c9df40 000007fef974a7f3[Gcframe:0000000030c9df40]0000000030c9dfc8 000007fef974a7f3[Gcframe:0000000030c9dfc8]0000000030c9e130 000007fef974a7f3[gcframe:0000000030c9e130]0000000030c9e5c8 000007fef974a7f3[Helpermethodframe:0000000030c9e5c8]0000000030c9e6f0 000007FE9A1774FA * * * error:module load completed but symbols could not being loaded for ORACLE.DATAACC Ess.dllOracle.DataAccess.Client.OracleTuningAgent.DoScan ()0000000030c9e740 000007fe9a1688e5  Oracle.DataAccess.Client.OracleTuningAgent.TuningFunction ()0000000030c9e7c0 000007fef86339a5 System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 0000000030c9e920 000007fef8633719 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 0000000030c9e950 000007fef86336f7 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback,System.Object) 0000000030c9e9a0 000007fef864adc1 System.Threading.ThreadHelper.ThreadStart () 0000000030c9ecb8 000007FEF974A7F3[Gcframe:0000000030c9ecb8]0000000030c9f008 000007fef974a7f3[debuggeru2mcatchhandlerframe:0000000030c9f008]0000000030c9f198 000007fef974a7f3[contexttransitionframe:0000000030c9f198]0000000030c9f3b8 000007fef974a7f3[Debuggeru2mcatchhandlerframe:0000000030c9f3b8]OS Thread Id:0x20bd4 (61) Child SP IP call site00000000031bdc58 00000000779cd96a[helpermethodframe:00000000031bdc58]System.AppDomain.nUnload (Int32) 00000000031BDD50 000007fef8e3e892 System.AppDomain.Unload (System.AppDomain) 00000000031BDDB0 000007FEF4B0C4A2 * * * warning:unable to verify checksum for System.Web.ni.dllSystem.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain (System.Object) 00000000031bde30 000007fef86339a5 System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 00000000031bdf90 000007fef8633719 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 00000000031bdfc0 000007fef866216f System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () 00000000031be010 000007fef866136a System.Threading.ThreadPoolWorkQueue.Dispatch () 00000000031be578 000007FEF974A7F3[debuggeru2mcatchhandlerframe:00000000031be578]00000000031be708 000007fef974a7f3[contexttransitionframe:00000000031be708]00000000031be928 000007fef974a7f3[debuggeru2mcatchhandlerframe:00000000031be928]OS Thread Id: 0x3314 (  +)

Child SP IP call Site0000000003bee5b8 00000000779cd3fa[Helpermethodframe:0000000003bee5b8]0000000003bee6e0 000007fe9a1774fa Oracle.DataAccess.Client.OracleTuningAgent.DoScan () 0000000003bee730 000007fe9a1688e5 Oracle.DataAccess.Client.OracleTuningAgent.TuningFunction () 0000000003bee7b0 000007fef86339a5 System.Threading.ExecutionContext.RunInternal ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 0000000003bee910 000007fef8633719 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 0000000003bee940 000007fef86336f7 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback,System.Object) 0000000003bee990 000007fef864adc1 System.Threading.ThreadHelper.ThreadStart () 0000000003beeca8 000007FEF974A7F3[gcframe:0000000003beeca8]0000000003beeff8 000007fef974a7f3[Debuggeru2mcatchhandlerframe:0000000003beeff8]0000000003bef188 000007fef974a7f3[contexttransitionframe:0000000003bef188]0000000003bef3a8 000007fef974a7f3[debuggeru2mcatchhandlerframe:0000000003bef3a8]OS Thread id:0xa130 (63) Child SP IP call site0000000003ecdc48 00000000779cd96a[helpermethodframe:0000000003ecdc48]System.AppDomain.nUnload (Int32) 0000000003ecdd40 000007fef8e3e892 System.AppDomain.Unload (System.AppDomain) 0000000003ecdda0 000007fef4b0c4a2 System.Web.HttpRuntime.ReleaseResourcesAndUnloadAppDomain (System.Object) 0000000003ecde20 000007fef86339a5 System.Threading.ExecutionContext.RunInternal ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 0000000003ecdf80 000007fef8633719 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 0000000003ecdfb0 000007fef866216f System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () 0000000003ece000 000007fef866136a System.Threading.ThreadPoolWorkQueue.Dispatch () 0000000003ece568 000007FEF974A7F3[debuggeru2mcatchhandlerframe:0000000003ece568]0000000003ece6f8 000007fef974a7f3[Contexttransitionframe:0000000003ece6f8]0000000003ece918 000007fef974a7f3[debuggeru2mcatchhandlerframe:0000000003ece918]OS Thread id:0xe0c (64) Child SP IP call Site000000000528e918 00000000779CD3FA[helpermethodframe:000000000528e918]000000000528ea40 000007fe9a1774fa Oracle.DataAccess.Client.OracleTuningAgent.DoScan () 000000000528ea90 000007fe9a1688e5 Oracle.DataAccess.Client.OracleTuningAgent.TuningFunction () 000000000528EB10 000007fef86339a5 System.Threading.ExecutionContext.RunInternal ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 000000000528ec70 000007fef8633719 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object,Boolean) 000000000528eca0 000007fef86336f7 System.Threading.ExecutionContext.Run ( System.Threading.ExecutionContext, System.Threading.ContextCallback,System.Object) 000000000528ecf0 000007fef864adc1 System.Threading.ThreadHelper.ThreadStart () 000000000528f008 000007FEF974A7F3[gcframe:000000000528f008]000000000528f358 000007fef974a7f3[debuggeru2mcatchhandlerframe:000000000528f358]000000000528f4e8 000007fef974a7f3[Contexttransitionframe:000000000528f4e8]000000000528f708 000007fef974a7f3[debuggeru2mcatchhandlerframe:000000000528f708]

Other precautions:

BulkCopy the code that triggered the bug, use Oracle's bulkcopy with caution.

Reference information:

Http://www.cnblogs.com/zhaoguan_wang/p/5505751.html

Http://www.cnblogs.com/xling/p/4347165.html

Analysis Case: Application Server w3wp process CPU lasts more than 90% (Oracle client Bug)