"Performance Diagnosis" XI. comprehensive analysis of performance problems (case 2,windbg, Wireshark)

"Problem description":

Previous time there is a project feedback, commonly used approval function sometimes slow response, multiple administrator features irregular client unresponsive situation, and once the administrator function is stuck, it will also affect the ordinary business users so that many users can not operate. The system is for mixed scenarios, the administrator function uses the CS client, and the business user is using the Web.

"Environmental Information":

Application Server: Windows Server R2 IIS7. NET 4.0

Database server: AIX Oracle 10.2.0.4

"Process":

Two cases have been traced:

1. Indeed, some of the documents awaiting review are slow to reproduce and appear to be related to documents (slow documents are slow to open repeatedly, other documents have no similar problems) and feel strange;
2. After the administrator function is not responding to the client, look at the performance view of the corresponding database side, there is no SQL blocking information, there is no obvious execution time is very long, inefficient SQL;

When a system hangs, the dump file of the application server is crawled and a noticeable thread block is found:

1. Line Line 105 Blocks the other 4 threads {(9-1)/2}, while the Line 105 thread waits for the database to return, OracleLob, which is a special data type that needs to be alerted;
2. According to the thread stack: operatedatabasemanager.getpacekage (System.String), we can quickly locate the program code and corresponding SQL script;
3. As you can see from WinDbg, other threads are blocked on the Metadatacache.getobject () method, looking at the corresponding source code and discovering that the method uses a static object as the synchronization lock object;

is a thread that blocks the log log for the corresponding WinDbg

Microsoft (R) Windows Debugger Version 6.12.0002.633 amd64copyright (c) Microsoft Corporation. All rights reserved. Loading Dump File [D:\dumps\w3wp. Dmp]user Mini Dump File with full memory:only application data is Availablesymbol search path is:srv*c:\symbols*http://m Sdl.microsoft.com/download/symbolsexecutable Search Path Is:windows 7 Version 7601 (Service Pack 1) MP (8 procs) free x64 Product:server, Suite:enterprise Terminalserver singleusertsmachine name:debug session time:tue APR 14 14:22:10.000 201 5 (UTC + 8:00am) System uptime:19 days 20:26:50.903process uptime:0 days 5:33:15.000 ..... ..... ....... ............... ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .......... ................... ....... ............................. ntdll! ntwaitforsingleobject+0xa:00000000 ' 7711135a c3 ret0:000>. Load d:\dumps\sos.dll0:000>!tpcpu utilization:0%worker thread:total:29 running:6 idle:23 MaxLimit: Minlimit:8work Request in queue:0--------------------------------------number of timers:43---------------------- ----------------Completion Port thread:total:1 free:1 maxfree:16 currentlimit:0 maxlimit:800 minlimit:80:000>!syncblkindex syncblock monitorheld recursion owning Thread Info SyncBlock Owner 47 00000 00005532488 9 1 000000001385b1e0 93a0 000000017FFEFBE0 System.Object----------------------------- Total 6125CCW 4RCW 11ComClassFactory 0Free 43330:000> ~105sntdll! ntwaitforsingleobject+0xa:00000000 ' 7711135a c3 ret 0:105>!clrstack   OS Thread id:0x93a0 () child SP IP call site000000000fbac038 000000007711135a [ndirectmethodframe STANDALONE:000000000FBAC038] System.Data.Common.UnsafeNativeMethods.OCILobRead ( System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, UInt32 ByRef, UInt32, IntPtr, UInt32, IntPtr, IntPtr, UInt16, charsetform) 000000000fbabfb0 000007fef3455640 Domainneutralilstubclass.il_stub_pinvoke (System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle , System.Data.OracleClient.OciHandle, UInt32 ByRef, UInt32, IntPtr, UInt32, IntPtr, IntPtr, UInt16, charsetform) * * * Warni Ng:unable to verify checksum for System.Data.OracleClient.ni.dll000000000fbac0f0 000007fef344ea9b System.Data.OracleClient.TracedNativeMethods.OCILobRead (System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, System.Data.OracleClient.OciHandle, Int32 ByRef, UInt32, IntPtr, UInt32, UInt16, Charsetform)000000000fbac1d0 000007fef347c63d System.Data.OracleClient.OracleLob.Read(byte[], Int32, Int32)000000000fbac2a0 000007fef347d483 System.Data.OracleClient.OracleLob.get_Value ()000000000fbac320 000007fef34721b6 System.Data.OracleClient.OracleColumn.GetValue ( System.Data.OracleClient.NativeBuffer_RowBuffer) 000000000fbac3d0 000007fef346efd2 System.Data.OracleClient.OracleDataReader.GetValues (system.object[]) 000000000fbac420 000007FEDD16FAEC System.Data.ProviderBase.SchemaMapping.LoadDataRow () * * * warning:unable to verify checksum for System.Data.ni.dll000000000fbac460 000007FEDD1674FD System.Data.Common.DataAdapter.FillLoadDataRow ( System.Data.ProviderBase.SchemaMapping) 000000000fbac4f0 000007fedd1673d4 System.Data.Common.DataAdapter.FillFromReader (System.Data.DataSet, System.Data.DataTable, System.String, System.Data.ProviderBase.DataReaderContainer, Int32, Int32, System.Data.DataColumn, System.Object) 000000000fbac5b0 000007fedd167271 System.Data.Common.DataAdapter.Fill (System.Data.DataSet, System.String, System.Data.IDataReader, Int32, Int32) 000000000fbac670 000007fedd167ef6 System.Data.Common.DbDataAdapter.FillInternal (System.Data.DataSet, System.Data.DataTablE[], Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior) 000000000fbac720 000007FEDD167CC4 System.Data.Common.DbDataAdapter.Fill (System.Data.DataSet, Int32, Int32, System.String, System.Data.IDbCommand, System.Data.CommandBehavior)000000000fbac7d0 000007fedd167ac2 System.Data.Common.DbDataAdapter.Fill(System.Data.DataSet) 000000000fbac880 000007ff002393ee xxxxxx. Platform.Core.DataAccess.Database.GetDataSet (System.String, System.String, system.data.idbdataparameter[], Boolean) * * * * error:module load completed but symbols could not being loaded for xxxxxx. Platform.Core.DataAccess.dll000000000fbac950 000007ff002391cd xxxxxx. Platform.Core.DataAccess.Database.GetDataSet (System.String, system.data.idbdataparameter[], Boolean) 000000000fbac990 000007ff0023906a xxxxxx. Platform.Core.DataAccess.Database.ExecuteDataSet (System.String, system.data.idbdataparameter[]) 000000000fbac9f0 000007ff00238810 xxxxxx. Platform.Core.DataAccess.Database.ExecuteDataSet (System.String, system.object[]) 000000000fbaca60 000007ff01566749 xxxxxx. Platform.Workflow.Server.Designer.OperateDataBaseManager.GetPacekage (System.String)  Warning:unable to verify checksum for xxxxxx. platform.workflow.server.dll*** error:module load completed but symbols could is loaded for xxxxxx. Platform.Workflow.Server.dll000000000fbacab0 000007ff01566548 xxxxxx. Platform.Workflow.Server.Designer.OperateDataBaseManager.Load (System.String) 000000000fbacb00 000007ff0024bbfe Xxxxxx. Platform.Resource.ResourceManager.MetadataService.GetCheckedinVersionContent (System.String, System.Data.DataRow , System.String ByRef) * * * Error:module load completed but symbols could is loaded for xxxxxx. Platform.Resource.MetadataManager.dll000000000fbacb60 000007ff0024b842 xxxxxx. Platform.Resource.ResourceManager.MetadataService.GetMetadataContentString (System.String, System.String ByRef, System.String) 000000000FBACBE0 000007ff0024b53b xxxxxx. Platform.Resource.ResourceManager.MetadataService.GetMetadataContentFromDB (System.String) 000000000fbacc40 000007FF00243D1A xxxxxx. Platform.Resource.ResourceManager.MetadataCache.GetObject (System.String, xxxxXx. Platform.AppFramework.Service.xxxState) 000000000fbaccb0 000007ff00fb036b xxxxxx. Platform.Resource.MetadataManager.MetadataRestFul.TryGetNewVersionContent (System.IO.BinaryReader, System.IO.BinaryWriter) 000000000fbad418 000007fef98d10b4 [debuggeru2mcatchhandlerframe:000000000fbad418] 000000000fbad500 000007fef98d10b4 [customgcframe:000000000fbad500] 000000000fbad4c8 000007fef98d10b4 [GCFrame: 000000000FBAD4C8] 000000000fbad458 000007fef98d10b4 [gcframe:000000000fbad458] 000000000fbad828 000007fef98d10b4 [ helpermethodframe_protectobj:000000000fbad828] System.runtimemethodhandle._invokemethodfast ( System.iruntimemethodinfo, System.Object, system.object[], system.signaturestruct ByRef, System.Reflection.MethodAttributes, System.runtimetype) 000000000fbad970 000007fef895587f System.Reflection.RuntimeMethodInfo.Invoke (System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, system.object[], System.Globalization.CultureInfo, Boolean) * * * warning:unable to verify Checksum For mscorlib.ni.dll000000000fbadab0 000007fef89a0426 System.Reflection.RuntimeMethodInfo.Invoke (System.Object, System.Reflection.BindingFlags, System.Reflection.Binder, system.object[], System.Globalization.CultureInfo) 000000000fbadb00 000007ff00b4f3b3 xxxxxx. Platform.AppFramework.Service.xxxRestfulContext.InternalInvoke (System.String, System.Reflection.MethodInfo, System.Object, system.object[], int32[] ByRef, system.object[] ByRef) * * * Error:module load completed but symbols could no T is loaded for xxxxxx. Platform.AppFramework.RestfulService.dll000000000fbadb60 000007ff00b4ebf0 xxxxxx. Platform.AppFramework.Service.xxxRestfulContext.Invoke (System.String, System.String, System.String, Boolean, System.object[], int32[] ByRef, system.object[] byref) 000000000fbadc40 000007ff00c28ae8 xxxxxx. Platform.AppFramework.Service.xxxRestfulContext.Invoke (System.String, System.String, System.String, Boolean, System.object[]) 000000000fbadca0 000007ff00c2876b xxxxxx. Platform.AppFramework.RESTFulWebServiCe.xxxHttpWebHandler.InvokeStream (System.IO.BinaryReader, System.Web.HttpContext, Boolean) * * * error:module load Completed but symbols could is not being loaded for xxxxxx. Platform.AppFramework.RESTFulWebService.dll000000000fbadd80 000007ff00b4de0c xxxxxx. Platform.AppFramework.RESTFulWebService.xxxHttpWebHandler.ProcessRequest (System.Web.HttpContext) 000000000fbade70 000007fef1eb5c25 system.web.httpapplication+ CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute () * * * warning:unable to verify Checksum For SYSTEM.WEB.NI.DLL000000000FBADF30 000007fef1e8337a System.Web.HttpApplication.ExecuteStep (IExecutionStep, Boolean ByRef) 000000000fbadfe0 000007fef1e97030 system.web.httpapplication+applicationstepmanager.resumesteps ( System.Exception) 000000000fbae0a0 000007fef1e82879 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest (System.Web.HttpContext, System.AsyncCallback, System.Object) 000000000fbae100 000007fef1e8777c System.Web.HttpRuntime.ProcessRequestInternal (System.Web.HttpWorkerRequest) 000000000fbae180 000007fef1e86477 System.Web.Hosting.ISAPIRuntime.ProcessRequest (INTPTR, Int32) 000000000fbae2b0 000007FEF25ED6DD DOMAINNEUTRALILSTUBCLASS.IL_STUB_COMTOCLR (Int64, Int32, Int32 ByRef) 000000000fbae5e8 000007fef993e93e [gcframe: 000000000fbae5e8] 000000000fbae628 000007fef993e93e [contexttransitionframe:000000000fbae628] 000000000fbae668 000007fef993e93e [gcframe:000000000fbae668] 000000000fbae850 000007fef993e93e [commethodframe:000000000fbae850]

 0:105> ~* e!clrstackOS Thread ID:0X2AEC (0) Unable to walk the managed stack. The current thread was likely not a managed thread. You can run!threads to get a list of managed threads inthe processOS Thread id:0x15f0 (106)
Child SP IP Call Site

000000000ed5df70 00000000771118ca [Gcframe:000000000ed5df70]
000000000ed5e1c8 00000000771118ca [Gcframe:000000000ed5e1c8]
000000000ed5e178 00000000771118ca [helpermethodframe:000000000ed5e178] System.Threading.Monitor.Enter (System.Object)
000000000ed5e2d0 000007ff00243ce8 xxxxxx. Platform.resource.ResourceManager.MetadataCache.GetObject(System.String, xxxxxx. Platform.AppFramework.Service.GSPState)
000000000ed5e340 000007ff00d40c8d xxxxxx. Platform.Resource.ResourceManager.DefaultValueManager.IsFormMobileForm (System.String)
000000000ed5e380 000007ff00d40ac0 xxxxxx. Platform.Resource.ResourceManager.DefaultValueManager.GetDefaultValueXmlExactly (System.String, System.String, System.String)
000000000ed5e3d0 000007ff00d409df xxxxxx. PLATFORM.ENGINE.DATAACCESS.COMMON.DAEUTILS.GETDEFAULTVALUEXML (System.String, System.String, System.String)
000000000ed5e420 000007ff00d40583 xxxxxx. Platform.Engine.DataAccess.GSPDataAccessEngine.NewObjectWithDefault (xxxxxx. Platform.Resource.Metadata.DataModel.GspDataModel, System.String, System.String)
000000000ed5e500 000007ff0091495d xxxxxx. Platform.BizComponent.JSProxy.BizComponentRESTFulService.InvokeInternal (System.String, system.object[], System.Collections.Generic.Dictionary ' 2<int32,system.object> ByRef, Boolean ByRef)
000000000ed5e590 000007ff0023fe16 xxxxxx. Platform.BizComponent.JSProxy.BizComponentRESTFulService.Get (System.Collections.Generic.Dictionary ' 2< system.string,system.string>)
000000000ed5e610 000007ff002276d8 xxxxxx. Platform.WebFramework.RestFulService.RESTFulHandlerForWeb.ProcessRequest (System.Web.HttpContext)
000000000ed5e840 000007fef1eb5c25 system.web.httpapplication+ CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute ()
000000000ed5e900 000007fef1e8337a System.Web.HttpApplication.ExecuteStep (IExecutionStep, Boolean ByRef)
000000000ed5e9b0 000007fef1e97030 System.web.httpapplication+applicationstepmanager.resumesteps (System.Exception )
000000000ed5ea70 000007fef1e82879 System.Web.HttpApplication.System.Web.IHttpAsyncHandler.BeginProcessRequest ( System.Web.HttpContext, System.AsyncCallback, System.Object)
000000000ed5ead0 000007fef1e8777c System.Web.HttpRuntime.ProcessRequestInternal (System.Web.HttpWorkerRequest)
000000000ed5eb50 000007fef1e86477 System.Web.Hosting.ISAPIRuntime.ProcessRequest (IntPtr, Int32)
000000000ed5ec80 000007FEF25ED6DD domainneutralilstubclass.il_stub_comtoclr (Int64, Int32, Int32 ByRef)
000000000ed5efb8 000007fef993e93e [GCFRAME:000000000ED5EFB8]
000000000ed5eff8 000007fef993e93e [CONTEXTTRANSITIONFRAME:000000000ED5EFF8]
000000000ed5f038 000007fef993e93e [gcframe:000000000ed5f038]
000000000ed5f220 000007fef993e93e [commethodframe:000000000ed5f220]

To block the source line Line 105 (blocked by the database return), execute the SQL corresponding program code:

The Synchronizer code that is blocking the thread:

At this point it should be clear that threads that execute SQL for a long time block other threads, while thread synchronization locks are too granular (to prevent duplicate values from being inserted for data caching only, and the result locks all metadata objects). But is there really a problem with the SQL script?

"Network Analysis":

By putting the SQL into Plsql, the client responds a long time, and after testing the different documents, it is found that the problem occurs when the LOB type has more than 16K of data, but there is no problem with the SQL in the test environment. The database server was previously tracked and no problems were found, so the target began to move to the application server to the database server's network environment.

Direct access to the database directly using both ADO and JDBC has found a response latency issue. Communicating with the customer's IT staff that a Cisco firewall has been passed from the application server to the database. We are in the application server, application server-side switch, database server-side switch, database server, 4 points for network capture. After comparison, it was found that the data packets of two switches before and after the firewall were obviously problematic: there was a very obvious case of packet chaos, the number of packets was inconsistent, and the window size decreased, and all the network packets passing through the firewall, the value of window size was changed from 64k to 16k;

Such as:

After the situation is fed back to Cisco's agents, they adjust the firewall settings: It seems to be a traffic control instead of a special understanding.

At this point, the problem is completely solved!

"Performance Diagnosis" XI. comprehensive analysis of performance problems (case 2,windbg, Wireshark)