Website high CPU Analysis

After the website is upgraded, it is said that there is a high CPU problem. After the mstscjob is mounted, the CPU usage of w3wp.exe is almost full, and my remote desktop operations are also slow. Download windbg, install it, and capture two dump instances. First! Runaway,

0: 022>! Runaway
User Mode Time
Thread time
22: 8cc 0 days 0:17:15. 238
23: 4d8 0 days 0:15:20. 936
15: 898 0 days 0:09:15. 316
24: C64 0 days 0:05:07. 587
This is the first four threads, which took about 47 minutes. Let's look at the time:

0: 022>. Time
Debug session time: Sun Mar 9 15:07:52. 000 2008 (GMT + 8)
System uptime: 1 days 2:04:59. 903
Process uptime: 0 days 0:55:38. 000
Kernel Time: 0 days 0:00:07. 000
User time: 0 days 0:48:41. 000

Cool,. Time and! Runaway is almost the same. Let's take a look! Threadpool Number:

0: 022>! Threadpool
CPU utilization 100%
Worker thread: Total: 5 running: 5 idle: 0 maxlimit: 200 minlimit: 2
Work request in queue: 15
Unknown Function: 6a2aa6fb context: 0127c5b8
Asynctimercallbackcompletion timerinfo @ 01969b30
Asynctimercallbackcompletion timerinfo @ 01969428
Unknown Function: 6a2aa6fb context: 01_ccb8
Asynctimercallbackcompletion timerinfo @ 019698a8
Asynctimercallbackcompletion timerinfo @ 01969c98
Asynctimercallbackcompletion timerinfo @ 019f5778
Asynctimercallbackcompletion timerinfo @ 01918658
Asynctimercallbackcompletion timerinfo @ 01969788
Unknown Function: 6a2aa6fb context: 0127cb38
Asynctimercallbackcompletion timerinfo @ 019695d8
Unknown Function: 6a2aa6fb context: 0127caf8
Asynctimercallbackcompletion timerinfo @ 01969740
Unknown Function: 6a2aa6fb context: 0127c178
Unknown Function: 6a2aa6fb context: 0127cbf8
--------------------------------------
Number of Timers: 9
--------------------------------------
Completion port thread: Total: 2 free: 2 maxfree: 4 Currentlimit: 2 maxlimit: 200 minlimit: 2

This information is enough, and the culprit is! Let's take a look at the several runaway threads:

0: 022> KB
Childebp retaddr ARGs to child
00005205 7a4c72c0 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexcharclass. charinclassrecursive (char, system. String, int32) + 0x20
00005205 7a4b16e5 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexinterpreter. Go () + 0xc40
00005205 7a4b15c3 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexrunner. Scan (system. Text. regularexpressions. RegEx, system. String, int32, int32, int32, int32, Boolean) + 0xa5
00005205 7a4b14af 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Run (Boolean, int32, system. String, int32, int32, int32) + 0x103
Ffffffff 7a4dc313 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Match (system. String, int32) + 0x1f
Ffffffff 7a4d1c39 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexreplacement. Replace (system. Text. regularexpressions. RegEx, system. String, int32, int32) + 0x47
0606f8ac 7a4d1b53 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Replace (system. String, system. String, int32, int32) + 0x99
05d8ed04 0ff965c7 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Replace (system. String, system. String) + 0x2f
05d8ed04 0ff96530 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Text. htmlhelper. removehtmlcomments (system. String) + 0x67
00000000 0ff96477 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Text. htmlhelper. removehtmlandcomments (system. String) + 0x38
10c3e8a8 0ff96142 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. components. aggregatedpost. set_description (system. String) + 0x2f
10c3e8a8 0ff95d8e 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Data. datahelper. loadaggreatedpost (system. Data. idatareader) + 0x1e2
10c3e910 0ff95c02 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Data. databaseobjectprovider. getaggregatedposts (int32, int32, int32) + 0x126
10c3e958 0ff9e4eb 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Data. cacher. getpagedaggregatedposts (int32, int32, int32, 3rdparty. Framework. cacheduration, Boolean) + 0x14a
10c3e9b0 0ff9e364 00000000 00000000 00000000 3rdparty_blog_web! 3rdparty. Blog. Web. webform1.binddata () + 0x16b
10c3ec04 66f12980 00000000 00000000 00000000 3rdparty_blog_web! 3rdparty. Blog. Web. webform1.page _ load (system. Object, system. eventargs) + 0x44
10c3ec04 6628eff2 00000000 00000000 00000000 system_web_regularexpressions_ni! System. Web. util. callihelper. eventargfunctioncaller (intptr, system. Object, system. Object, system. eventargs) + 0x10
10c3ec04 6613cb24 00000000 00000000 00000000 system_web_ni! System. Web. util. callieventhandlerdelegateproxy. Callback (system. Object, system. eventargs) + 0x22
10c3ec04 6613cb70 00000000 00000000 00000000 system_web_ni! System. Web. UI. Control. onload (system. eventargs) + 0x64
10c3ec04 6614e14d 00000000 00000000 00000000 system_web_ni! System. Web. UI. Control. loadrecursive () + 0x30

Let's look at other threads:

0: 015> KB
Childebp retaddr ARGs to child
2017515e 7a4b16e5 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexinterpreter. Go () + 0x152b
2017515e 7a4b15c3 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexrunner. Scan (system. Text. regularexpressions. RegEx, system. String, int32, int32, int32, int32, Boolean) + 0xa5
2017515e 7a4b14af 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Run (Boolean, int32, system. String, int32, int32, int32) + 0x103
Ffffffff 7a4dc313 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Match (system. String, int32) + 0x1f
Ffffffff 7a4d1c39 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. regexreplacement. Replace (system. Text. regularexpressions. RegEx, system. String, int32, int32) + 0x47
05f79678 7a4d1b53 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Replace (system. String, system. String, int32, int32) + 0x99
05db3568 0ff965c7 00000000 00000000 00000000 system_ni! System. Text. regularexpressions. RegEx. Replace (system. String, system. String) + 0x2f
05db3568 0ff96530 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Text. htmlhelper. removehtmlcomments (system. String) + 0x67
00000000 0ff96477 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Text. htmlhelper. removehtmlandcomments (system. String) + 0x38
0140f0d8 0ff96142 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. components. aggregatedpost. set_description (system. String) + 0x2f
0140f0d8 0ff95d8e 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Data. datahelper. loadaggreatedpost (system. Data. idatareader) + 0x1e2
0140f140 0ff95c02 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Data. databaseobjectprovider. getaggregatedposts (int32, int32, int32) + 0x126
0140f188 0ff9e4eb 00000000 00000000 00000000 3rdparty_framework! 3rdparty. Framework. Data. cacher. getpagedaggregatedposts (int32, int32, int32, 3rdparty. Framework. cacheduration, Boolean) + 0x14a
0140f1e0 0ff9e364 00000000 00000000 00000000 3rdparty_blog_web! 3rdparty. Blog. Web. webform1.binddata () + 0x16b
0140f434 66f12980 00000000 00000000 00000000 3rdparty_blog_web! 3rdparty. Blog. Web. webform1.page _ load (system. Object, system. eventargs) + 0x44
0140f434 6628eff2 00000000 00000000 00000000 system_web_regularexpressions_ni! System. Web. util. callihelper. eventargfunctioncaller (intptr, system. Object, system. Object, system. eventargs) + 0x10
0140f434 6613cb24 00000000 00000000 00000000 system_web_ni! System. Web. util. callieventhandlerdelegateproxy. Callback (system. Object, system. eventargs) + 0x22
0140f434 6613cb70 00000000 00000000 00000000 system_web_ni! System. Web. UI. Control. onload (system. eventargs) + 0x64
0140f434 6614e14d 00000000 00000000 00000000 system_web_ni! System. Web. UI. Control. loadrecursive () + 0x30
0140f434 6614d8e3 00000000 00000000 00000000 system_web_ni! System. Web. UI. Page. processrequestmain (Boolean, Boolean) + 0x59d

The other two threads are similar to the two.

I feel so lucky to have such a simple high CPU. Haha! Why is RegEx. replace slow? Run it! Clrstack-a command:

0: 015>! Clrstack-
OS thread ID: 0x898 (15)
ESP EIP

================================ Others are too long, omitted ...... ==================================

0140f034 7a4d1b53 system. Text. regularexpressions. RegEx. Replace (system. String, system. String)
Parameters:
This = <no data>
Input = <no data>
Replacement = <no data>

0140f044 0ff965c7 joycode. Framework. Text. htmlhelper. removehtmlcomments (system. String)
Parameters:
TEXT = 0x05da9298
Locals:
<CLR Reg> = 0x05db3568
<CLR Reg> = 0x00000000
0x0140f044 = 0x00000000

================================ Others are too long, omitted ...... ==================================

Oh, the address of the string is 0x05da9298. Check the size. It is 41 K.

0: 015>! Objsize 0x05da9298
Sizeof (05da9298) = 41,680 (0xa2d0) bytes (system. String)

If you do this string, we can see that it is an HTML text. Because the data of runaway and. Time and threadpool is too obvious, so determines that the problem is that the replace string is too large or the frequency is too high.

After confirmation, removehtmlcommentsCodeThe RegEx. Replace in is useless. So, the comment is directly removed and the problem is solved.

Take a look at other things:

Address	Mt	Priority	Create Time	Expires	Last Updated	Key Class Name
01b1c6cc	6639d580	Normal	03/09/2008 06:12:14	12/31/9999 23:59:59 0	0	Dmachine/webroot/2/app_code system. Web. cachedpathdata
01b1f470	6639d580	Normal	03/09/2008 06:12:14	12/31/9999 23:59:59	0	Dmachine/webroot/2/global. asax system. Web. cachedpathdata
01b3b3b0	6639d580	Normal	03/09/2008 06:12:15	12/31/9999 23:59:59	03/09/2008 07:07:35	Dmachine/webroot/2/systemmessages system. Web. cachedpathdata

This is a small part of aspnetcache content. It is worth noting that there are a lot of cache content that expires on January 1, 9999. In fact, we have a lot of reason to judge that, "For A Long Time", w3wp will cause recycled problems due to out of memory, or the sudden increase in access traffic and sustained for a period of time, will cause the website to collapse in the short term.

Let's look at heap again:

0: 015>! Heapstat
heap gen0 gen1 gen2 LOH
heap0 7291624 255092 5071228 13381296
heap1 7178448 440428 4637748 14825768
total 14470072 695520 9708976

Free Space: Percentage
Heap0 84 12 6568 2011792 Soh: 0% LOH: 15%
Heap1 2588 12 392 7699728 Soh: 0% LOH: 51%
Total 2672 24 6960 9711520

Large objects are the largest, almost equal to the sum of gen0 + gen1 + gen2. Let's take a look at the overall situation first:

0: 015>. foreach (myobj {! Dumpheap-short-min 85000 }){! Objsize myobj}
Sizeof (09c07bf0) = 689,900 (0xa86ec) bytes (system. byte [])
Sizeof (09cb02e0) = 616,976 (0x96a10) bytes (system. byte [])
Sizeof (09e2f2a8) = 616,976 (0x96a10) bytes (system. byte [])
================================ Others are too long, omitted ...... ==================================
Sizeof (0c89cf50) = 616,976 (0x96a10) bytes (system. byte [])
Sizeof (09d7c320) = 131,084 (0x2000c) bytes (system. int32 [])
================================ Others are too long, omitted ...... ==================================
Sizeof (0bec8f60) = 1,114,124 (0x11000c) bytes (system. int32 [])
Sizeof (0c092a68) = 1,114,124 (0x11000c) bytes (system. int32 [])
Sizeof (09b8cb40) = 105,512 (0x19c28) bytes ()
Sizeof (09bc6780) = 263,280 (0x40470) bytes ()
================================ Others are too long, omitted ...... ==================================
Sizeof (0c42f080) = 4,644,560 (0x46ded0) bytes ()
Sizeof (09b000068) = 262,164 (0x40014) bytes (system. String)
================================ Others are too long, omitted ...... ==================================
Sizeof (0bda0f00) = 262,164 (0x40014) bytes (system. String)
Sizeof (0bfd8f70) = 262,164 (0x40014) bytes (system. String)

Byte array:

0: 015>! Gcroot 0c398670
Note: roots found On Stacks may be false positives. Run "! Help gcroot"
More info.
Scan thread 7 osthread C60
Scan thread 13 osthread AEC
Scan thread 14 osthread c68
Scan thread 15 osthread 898
ESP: 140f088: Root: 05da5d54 (system. Data. sqlclient. sqldatareader)->
05da5a18 (system. Data. sqlclient. sqlconnection)->
01d3a928 (system. Data. sqlclient. sqlinternalconnectiontds)->
05b84ccc (system. Data. providerbase. dbconnectionpool)->
05b84f9c (system. Collections. Generic. List '1 [[system. Data. providerbase. dbconnectioninternal, system. Data])->
05b84fb4 (system. object [])->
05e03e00 (system. Data. sqlclient. sqlinternalconnectiontds)->
05e03e9c (system. Data. sqlclient. tdsparser)->
05e03f3c (system. Data. sqlclient. tdsparserstateobject)->
0c398670 (system. byte [])
ESP: 140f0e8: Root: 05da5d54 (system. Data. sqlclient. sqldatareader)->
05da5a18 (system. Data. sqlclient. sqlconnection)->
01d3a928 (system. Data. sqlclient. sqlinternalconnectiontds)->
05b84ccc (system. Data. providerbase. dbconnectionpool)->
05b84f9c (system. Collections. Generic. List '1 [[system. Data. providerbase. dbconnectioninternal, system. Data])->
05b84fb4 (system. object [])->
05e03e00 (system. Data. sqlclient. sqlinternalconnectiontds)->
05e03e9c (system. Data. sqlclient. tdsparser)->
05e03f3c (system. Data. sqlclient. tdsparserstateobject)->
0c398670 (system. byte [])

Similar to other bye arrays, sqldatareader operations are performed. (Why? This callstack cannot be seen)

Check the int32 array:

0: 015>! Gcroot 09fd4e60
Note: roots found On Stacks may be false positives. Run "! Help gcroot"
More info.
Scan thread 7 osthread C60
================================ Others are too long, omitted ...... ==================================
Scan thread 24 osthread C64
ECX: Root: 01e51c1c (system. Text. regularexpressions. regexinterpreter)->
09fd4e60 (system. int32 [])
ESI: Root: 01e51c1c (system. Text. regularexpressions. regexinterpreter)->
09fd4e60 (system. int32 [])
ESP: 114ae598: Root: 01e51c1c (system. Text. regularexpressions. regexinterpreter)->
09fd4e60 (system. int32 [])
ESP: 114ae5ac: Root: 01e51c1c (system. Text. regularexpressions. regexinterpreter)->
09fd4e60 (system. int32 [])
Scan thread 26 osthread 9ac
Scan thread 27 osthread 80C

This is the regular expression analyzed above.

The string information, which is shown above, is HTML reply. For these types, we will make a summary (in all the large objects)

Byte array: 4 m, which is the data read by sqldatareader.

Int32 array: 5.8 m, which is caused by RegEx and the following string

String: 8.5 m

In addition:
0: 015>! Finddebugmodules
Loading all modules.
Searching for modules built in debug mode...

App_globalresources.-mywaxaw. dll not built release
App_code.fqhaeca6.dll not built release
3rdparty. Framework. dll not built release
App_global.asax.0y1c1i6p.dll not built release
3rdparty. Blog. Web. dll not built release
Freetextbox. dll not built release
3rdparty. blogml. dll not built release
3rdparty. extensibility. dll not built release
3rdparty. Installation. dll not built release
3rdparty. plugins. Core. dll not built release
3rdparty. plugins. Examples. dll not built release
3rdparty. Scripting. dll not built release
3rdparty. Web. Controls. dll not built release
App_web_lab8my1c.dll not built release
App_theme_home.clever. _ mjadjg. dll not built release
App_web_xppxn64l.dll not built release
App_web_yxlgarwj.dll not built release
App_web_a8wjdvdv.dll not built release
App_web_ijqvv7ky.dll not built release
App_web_xxkqxoen.dll not built release
App_Web_b25tsu-i.dll not built release
App_web_kcb_q9mh.dll not built release
App_theme_theme.elitecircle.jmea1orn.dll not built release
App_web_site.master.9fb3aa03.6svjtsyu.dll not built release
App_globalresources.-mywaxaw. Resources. dll not built release
App_theme_theme.default.rggbez76.dll not built release
App_web_post.aspx.9fb3aa03.ocnogniv.dll not built release
App_Web_k_ivm-xy.dll not built release
App_web_category.aspx.9fb3aa03.lq_3jsov.dll not built release
App_web_r8dwq35a.dll not built release
App_web_archivemonth.aspx.9fb3aa03.jkc9sbgz.dll not built release
App_Web_gallery.aspx.9fb3aa03.vc-q4rsf.dll not built release
App_web_contact.aspx.9fb3aa03.ayhi_kzk.dll not built release
App_Web_vocni5-f.dll not built release
App_web_zgfpt_vo.dll not built release

Done seaching

All the above modules are in debug mode. here we need to modify web. config and set DEBUG = false and batch compilation to true, which can also improve performance to a certain extent.

There is no conclusion, so it is over.