Catch the Worm (iv) thread safety caused by highcpu

A friend QQ group said that the website will be high CPU after the start, want to help look at the dump.

1. Open WinDbg load dump file after the first command LMF, this command shows the loaded DLL and path, this way can find a DLL to help load SOS, (well, I can't remember the length of the load path AH)

0:000> LMF

Start End Module Name

......

00007FFB ' a3750000 00007ffb ' a375a000 version C:\Windows\System32\version.dll

00007FFB ' a37e0000 00007ffb ' a444e000 system_ni C:\Windows\assembly\NativeImages_v4.0.30319_64\System\ 10e3367d9097070044e05c2825a4a1e9\system.ni.dll

00007FFB ' a4450000 00007ffb ' a457f000 Clrjit C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll

00007FFB ' a4580000 00007ffb ' a5b0d000 mscorlib_ni C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\ A20cafac04a2e9b3bcb5ec4d674775e5\mscorlib.ni.dll

......

2. Or Loadby is better than load (hehe)

0:000>. Loadby SOS Clrjit

We're going to look at the CPU.

0:000>!TP

CPU utilization:100%

Worker thread:total:8 running:7 idle:1 maxlimit:32767 minlimit:4

Work Request in queue:0

--------------------------------------

Number of timers:0

--------------------------------------

Completion Port thread:total:1 free:1 maxfree:8 currentlimit:1 maxlimit:1000 minlimit:4

Well, the situation is enough.

3. Let's see how high that thread is.

0:000>!runaway

User Mode Time

Thread time

21:32e4 0 Days 0:00:43.671

20:2d1c 0 Days 0:00:43.187

22:1110 0 Days 0:00:42.828

25:257c 0 Days 0:00:42.625

19:337c 0 Days 0:00:36.984

26:32c8 0 Days 0:00:35.906

30:2f78 0 Days 0:00:35.796

7:30C0 0 Days 0:00:00.140

11:2758 0 Days 0:00:00.062

13:2644 0 Days 0:00:00.031

12:2770 0 Days 0:00:00.031

。。。

4. Threads are too few, as if they are not GC-related, are user-state.

0:000>!threads

Threadcount:18

unstartedthread:0

Backgroundthread:17

pendingthread:0

Deadthread:1

Hosted Runtime:no Lock

ID OSID Threadobj State GC Mode GC Alloc Context Domain Count Apt Exception

7 1 30c0 000000df603e3170 28220 preemptive 0000000000000000:0000000000000000 000000df603d9ff0 0 Ukn

2 26c4 000000df6040e790 2b220 preemptive 000000e060f2c908:000000e060f2dc90 000000df603d9ff0 0 MTA (Finalizer)

3 2CAC 000000df5fc17e40 102a220 preemptive 0000000000000000:0000000000000000 000000df603d9ff0 0 MTA (Threadpool Worker )

4 33c0 000000df6047c7d0 21220 preemptive 0000000000000000:0000000000000000 000000df603d9ff0 0 Ukn

6 3174 000000df6051b420 1020220 preemptive 0000000000000000:0000000000000000 000000df603d9ff0 0 Ukn (Threadpool Worker )

7 337c 000000df60549fa0 1029220 Cooperative 000000e260b93590:000000e260b947d0 000000df6047b7d0 1 MTA (Threadpool Worke R

8 2d1c 000000df6054b370 1029220 Cooperative 000000e060fa2dd8:000000e060fa3c90 000000df6047b7d0 0 MTA (Threadpool Worke R

9 32e4 000000df60560a10 1029220 Cooperative 000000e260b5f130:000000e260b607d0 000000df6047b7d0 0 MTA (Threadpool Worke R

1110 000000df60561690 1029220 Cooperative 000000df60b5ab48:000000df60b5b178 000000df6047b7d0 0 MTA (Threadpool work Er

257c 000000df5fc28520 1029220 Cooperative 000000e160ae4988:000000e160ae6190 000000df6047b7d0 0 MTA (Threadpool work Er

32C8 000000df5fc29c90 1029220 Cooperative 000000e1610d4570:000000e1610d4a60 000000df6047b7d0 0 MTA (Threadpool work Er

XXXX 0 000000df5fc2a460 8039820 preemptive 0000000000000000:0000000000000000 000000df603d9ff0 0 Ukn (Threadpool Comple tion Port)

1a64 000000df5fc2b400 202b220 preemptive 0000000000000000:0000000000000000 000000df6047b7d0 1 MTA

2FBC 000000df5fc2bbd0 8029220 preemptive 000000df60f077d0:000000df60f09178 000000df603d9ff0 0 MTA (Threadpool Compl Etion Port)

2f78 000000df5fc28cf0 1029220 Cooperative 000000df6167e630:000000df6167f8b0 000000df6047b7d0 0 MTA (Threadpool work Er

5 312c 000000df5fc294c0 1029220 preemptive 000000e1610e0d88:000000e1610e2a60 000000df603d9ff0 0 MTA (Threadpool Worker )

2 2fb8 000000df5fc2ac30 20220 preemptive 000000e061adf4b8:000000e061ae1358 000000df603d9ff0 0 Ukn

3 2c88 000000df604c6f60 20220 preemptive 000000df6167fa30:000000df616818b0 000000df603d9ff0 0 Ukn

0:000>. Time

Debug session time:wed Sep 15:11:53.000 (UTC + 8:00)

System uptime:0 days 5:43:30.998

Process uptime:0 days 0:01:16.000

Kernel time:0 days 0:00:00.000

User time:0 days 0:04:42.000

5. Look for a few threads.

0:000> ~21s

MSCORLIB_NI+0X5535E0:

00007FFB ' a4ad35e0 488B4B10 mov rcx,qword ptr [rbx+10h] ds:000000e0 ' 607f0e28=000000e060fa2d78

0:021>!clrstack

OS Thread id:0x32e4 (21)

Child SP IP Call Site

000000e3ab54da90 00007ffba4ad35e0 System.Collections.Generic.Dictionary ' 2[[system.__canon, mscorlib],[system.__ Canon, mscorlib]]. Insert (System.__canon, System.__canon, Boolean)

000000E3AB54DB20 00007FFB46EC5DE9 * * * error:module load completed but symbols could not being loaded for System.Web.Optimiza Tion.dll

SYSTEM.WEB.OPTIMIZATION.BUNDLECOLLECTION.ADD (System.Web.Optimization.Bundle)

/********* here is the customer code *************/

000000e3ab54e400 00007ffb46a55aca Rabbit.Web.Environment.Impl.WebHost.ActivateShell ( Rabbit.Kernel.Environment.ShellBuilders.ShellContext)

000000e3ab54e490 00007ffb46a44fd3 Rabbit.web.environment.impl.webhost.<createandactivateshells>b__f ( Rabbit.Kernel.Environment.Configuration.ShellSettings)

000000e3ab54e520 00007ffba530df6d SYSTEM.THREADING.TASKS.PARALLEL+<>C__DISPLAYCLASSF ' 1[[System.__Canon, Mscorlib]].<forworker>b__c ()

000000e3ab54e630 00007ffba52f0452 System.Threading.Tasks.Task.InnerInvokeWithArg (System.Threading.Tasks.Task)

000000e3ab54e660 00007ffba54a8911 system.threading.tasks.task+<>c__displayclass11.< Executeselfreplicating>b__10 (System.Object)

000000e3ab54e700 00007ffba4ab30ee System.Threading.Tasks.Task.Execute ()

000000e3ab54e770 00007ffba4a18355 System.Threading.ExecutionContext.RunInternal ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

000000e3ab54e8d0 00007ffba4a180c9 System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

000000e3ab54e900 00007ffba4ab33c5 System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task BYREF)

000000e3ab54e9e0 00007ffba4ab2a65 System.Threading.Tasks.Task.ExecuteEntry (Boolean)

000000E3AB54EA20 00007ffba4a9a22a System.Threading.ThreadPoolWorkQueue.Dispatch ()

000000e3ab54ef78 00007ffba5c40453 [debuggeru2mcatchhandlerframe:000000e3ab54ef78]

000000e3ab54f118 00007ffba5c40453 [contexttransitionframe:000000e3ab54f118]

000000e3ab54f338 00007ffba5c40453 [debuggeru2mcatchhandlerframe:000000e3ab54f338]

0:021> ~20s

MSCORLIB_NI+0X5535F4:

00007FFB ' A4ad35f4 488d1476 Lea Rdx,[rsi+rsi*2]

0:020>!clrstack

OS Thread id:0x2d1c (20)

Child SP IP Call Site

000000E3AB40DBE0 00007ffba4ad35f4 System.Collections.Generic.Dictionary ' 2[[system.__canon, mscorlib],[system.__ Canon, mscorlib]]. Insert (System.__canon, System.__canon, Boolean)

000000e3ab40dc70 00007ffb46ec5de9 System.Web.Optimization.BundleCollection.Add (System.Web.Optimization.Bundle)

/********* here is the customer code *************/

000000e3ab40e5e0 00007ffb46a44fd3 Rabbit.web.environment.impl.webhost.<createandactivateshells>b__f ( Rabbit.Kernel.Environment.Configuration.ShellSettings)

000000e3ab40e670 00007ffba530df6d SYSTEM.THREADING.TASKS.PARALLEL+<>C__DISPLAYCLASSF ' 1[[System.__Canon, Mscorlib]].<forworker>b__c ()

000000e3ab40e780 00007ffba52f0452 System.Threading.Tasks.Task.InnerInvokeWithArg (System.Threading.Tasks.Task)

000000e3ab40e7b0 00007ffba54a8911 system.threading.tasks.task+<>c__displayclass11.< Executeselfreplicating>b__10 (System.Object)

000000e3ab40e850 00007ffba4ab30ee System.Threading.Tasks.Task.Execute ()

000000E3AB40E8C0 00007ffba4a18355 System.Threading.ExecutionContext.RunInternal ( System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

000000e3ab40ea20 00007ffba4a180c9 System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)

000000e3ab40ea50 00007ffba4ab33c5 System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task BYREF)

000000e3ab40eb30 00007ffba4ab2a65 System.Threading.Tasks.Task.ExecuteEntry (Boolean)

000000e3ab40eb70 00007ffba4a9a22a System.Threading.ThreadPoolWorkQueue.Dispatch ()

000000E3AB40F0C8 00007ffba5c40453 [DEBUGGERU2MCATCHHANDLERFRAME:000000E3AB40F0C8]

000000e3ab40f268 00007ffba5c40453 [contexttransitionframe:000000e3ab40f268]

000000e3ab40f488 00007ffba5c40453 [debuggeru2mcatchhandlerframe:000000e3ab40f488]

After a few take a few take a long time to see, it is like this.

In fact, I saw the first one, the heart click Stops, the reason is below

Http://blogs.msdn.com/b/tess/archive/2009/12/21/high-cpu-in-net-app-using-a-static-generic-dictionary.aspx

In fact, this problem happened to Wu Zhencao classmate also met http://www.cnblogs.com/wlflovenet/p/debugCpu100.html

Tidy up your mind: Check for high CPU issues

1. First look at the thread pool and thread run time, (we generally catch two dump to compare)
2. Make sure it's not a GC. Because the high CPU directly look at the stack (in fact, because the GC allocates memory what can also be seen, see the previous article catching insects (ii) GC-ledhang)
3. To see if there is a dead loop, there is not a lot of hash calculation, if it is ASP, see if there is a static container (Dict or List) of thread safety issues.

Catch the Worm (iv) thread safety caused by highcpu