windbg debug Analysis ASP. cpu100% problem

Tag: The for MEM symbol requires export call using the command frame path

In order to save costs, the company recently had a number of servers dropped configuration, the CPU from 8 to 2 cores. itself is a small site, access is not high, the CPU will always go up to 100% and can continue for a few hours, the direct force to end the process can be maintained for several hours, a few hours later, the operation of the side is always subject to the CPU warning SMS is very distressed, In a logical sense, even lowering the configuration will not keep the CPU 100%.

Share the experience of using WINDBG to find out CPU 100% issues below:

1. Create a dump file

　　　　

Note whether the process is 32-bit or 64-bit. 64-bit processes can be created directly, 32-bit processes should be created using the C:\Windows\SysWOW64\taskmgr.exe Task Manager, or they will be WinDbg in error.

2. Load the dump file with WinDbg

The 32-bit process uses x86 WinDbg to load, and 64 for the process to load using x64 's WinDbg. https://github.com/LonelyCodelang/Tools/tree/master/Windbg_x86_x64

　　

3. Configuring the Debug environment

When opened, it shows the environment in which the program was running, at which point the symbol file was not found:

Error:symbol file could not being found.  

If not configured, the use of the command will prompt the error, such as:

0:000>. Loadby SOS clrjit0:000>!tp*** error:symbol file could not being found.  defaulted to export symbols for Clr.dll-************* Symbol Loading Error Summary **************module name            ERRORCL R                    PDB not found:e:\appserver\symbols\dll\clr.pdb

E:\appserver is the directory where my DMP file is located, and it defaults to the symbols subdirectory to find the symbol file.

Then, configure the debugging environment under which this file is used.
In the menu command file->symbol path ... Open the dialog box, select Browse, find the directory of the DMP file directory related program Files E:\AppServer, the directory below the program-related exe,pdb file.

Enter the following command:

0:000>. sympath+ C:\symbolsSymbol Search Path is:e:\appserver;c:\symbolsexpanded Symbol search path is:e:\appserver; C:\symbolserror:attempts to access ' C:\Symbols ' failed:0x2-the system cannot find the file specified. Symbol Path Validation Summary **************response time                         (ms)     Locationok                                             

There is no control here, this folder can be generated later.

0:000> .symfix0:000>. symfix+ c:\symbols0:000>. Sympathsymbol Search Path is:srv*expanded Symbol search path is: srv*c:\symbols*http://msdl.microsoft.com/download/symbols************* Symbol Path Validation Summary ************* *response Time                         (ms)     locationdeferred  

That's right.
Execute the following reload command:

0:000>. Reload ......................................................................................................................... ...................

Then perform the following:

0:000>. Loadby SOS clr0:000>!tpthe version of SOS does not match the version of the CLR is debugging.  Pleaseload the matching version of SOS for the version of the CLR is debugging. CLR Version:4.0.30319.296sos version:4.6.96.0failed to load data access DLL, 0x80004005

It is suggested here that the version of SOS is more CLR mismatched, it is necessary to find the SOS.dll on the server where thedump file was generated, note that because the server program is 64-bit, it must be found in the. Net Framework64 directory and Mscordacwks.dll files are copied together (for the time being, we'll talk about them soon).

Just after the execution of this command, we were pleasantly surprised to find that the C:\Symbols directory created itself, and downloaded a few directories such as clr.pdb, this is the copy of the server just SOS.dll, Mscordacwks.dll, put on the local machine c:\symbols Directory below.

Execute the following commands again:

0:000>. Reload ......................................................................................................................... ................... 0:000>. Loadby SOS clr0:000>!tpthe version of SOS does not match the version of the CLR is debugging.  Pleaseload the matching version of SOS for the version of the CLR is debugging. CLR Version:4.0.30319.296sos version:4.6.96.0failed to load data access DLL, 0x80004005

Or an error, it appears that SOS.dll is not loaded correctly, with the following command:

0:000>. Load c:\symbols\sos.dll0:000>. loadby SOS clr0:000>!TP

Here the load command must take the path of SOS.dll. Loads it, and then executes the. loadby SOS CLR, which represents debugging. NET managed programs.

Start a long wait, the program window prompts:

*busy*
Downloading symbols for [clr.pdb]/

Wait until the relevant symbol files are all downloaded, and finally appeared a successful interface:

CPU utilization:11%worker thread:total:8 running:0 idle:8 maxlimit:32767 minlimit:8work Request in queue:0-------- ------------------------------Number of timers:14--------------------------------------completion Port Thread: Total:1 free:1 maxfree:16 currentlimit:0 maxlimit:1000 minlimit:8

　　4. Find the thread that consumes the most CPU and switch to thread change

Use !runaway to find out the CPU time consumed by each thread. The most time-consuming thread ID is 17, which is the thread that caused the cpu100%.

　　

　　~17s

switch to this thread

　　5. Use!clrstack to list the callstack of Thread 27

From here, we can locate the specific scheme, which is caused by the thread of the RABBITMQ consumption method, and then we can see the implementation of the internal method and then optimize it.

Related Reference articles:

https://blogs.msdn.microsoft.com/tess/2010/09/29/capturing-memory-dumps-for-32-bit-processes-on-an-x64-machine/

Https://www.cnblogs.com/bluedoctor/p/4813125.html

https://stackoverflow.com/questions/16422577/sos-does-not-support-the-current-target-architecture/16422887

windbg debug Analysis ASP. cpu100% problem