Windbg initial experience 3. continuously cutting the chain (outofmemory Case Study II)

The iron chain is always hard to cut, and the object reference chain of. NET is as strong as the iron chain. If you don't pay attention to it, you will make yourProgramCarry a heavy burden on your program, and then let your program crash. Next we will use windbg to analyze such an example.

As mentioned above, after resolving the cache problem of mongoe.net, the memory is like the flood water level for a long time, and the relentless surge immediately reaches the warning water level. There is no way to flood the flood. To completely solve the problem, you must rely on the windbg and dump files. This time, I captured a dump file that is about 1 GB. Okay, let's start over. View the drag tube heap:

0: 011>! Eeheap-GC
PDB symbol for mscorwks. dll not loaded
Number of GC heaps: 4
------------------------------
Heap 0 (000e22d0)
Generation 0 starts at 0x473a8500
Generation 1 starts at 0x4738941c
Generation 2 starts at 0x02860038
Ephemeral segment allocation context: None
Segment begin allocated size
1b84dc60 7b45382c 7b46a268 0x00016a3c (92732)
000f66d8 7a72c42c 7a74d308 0x00020edc (134876)
000eaef0 790d5588 790f4b38 0x0001f5b0 (128432)
02860000 02860038 0685cc24 0x03ffcbec (67095532)
30000000 30000038 33ffe93c 0x03ffe904 (67102980)
44350000 44350038 475e4afc 0x03294ac4 (53037764)
Large Object heap starts at 0x12860038
Segment begin allocated size
12860000 12860038 144620a8 0x01c02070 (29368432)
Fc100000 1fc00038 1fe659b0 0 0x00265978 (2513272)
Fc500000 50fc0038 5178d060 0x007cd028 (8179752)
Heap size 0xd91b88c (227653772)
------------------------------
Heap 1 (000e3538)
Generation 0 starts at 0x6e30f94c
Generation 1 starts at 0x6e2dad60
Generation 2 starts at 0x06860038
Ephemeral segment allocation context: None
Segment begin allocated size
06860000 06860038 0a85ff64 0x03ffff2c (67108652)
22000000 22000038 25ffffe0 0x03ffffa8 (67108776)
38350000 38350038 3c33b76c 0x03feb734 (67024692)
6d110000 6d110038 6e4fa678 0x013ea640 (20883008)
Large Object heap starts at 0x14860038
Segment begin allocated size
14860000 14860038 1489a178 0x0003a140 (237888)
Heap size 0xd40fd88 (222363016)
------------------------------
Heap 2 (000e4a48)
Generation 0 starts at 0x441e9d28
Generation 1 starts at 0x441d3978
Generation 2 starts at 0x0a860038
Ephemeral segment allocation context: None
Segment begin allocated size
0a860000 0a860038 0e85fb00 0x03fffac8 (67107528)
2a000000 2a000038 2dffd6a4 0x03ffd66c (67098220)
40350000 40350038 441f5d34 0x03ea5cfc (65690876)
Large Object heap starts at 0x16860038
Segment begin allocated size
16860000 16860038 16860048 0x00000010 (16)
Heap size 0xbea2e40 (199896640)
------------------------------
Heap 3 (000e5f30)
Generation 0 starts at 0x40224450
Generation 1 starts at 0x401e7bcc
Generation 2 starts at 0x0e860038
Ephemeral segment allocation context: None
Segment begin allocated size
0e860000 0e860038 1285f4b8 0x03fff133 (67105920)
26000000 26000038 29ffeee0 0x03ffeea8 (67104424)
3c350000 3c350038 4022445c 0x03ed4424 (65881124)
Large Object heap starts at 0x18860038
Segment begin allocated size
18860000 18860038 18b13eb0 0x002b3e78 (2834040)
Heap size 0xc1865c4 (202925508)
------------------------------
GC heap size 0x32d54a18 (852838936)

From the size of the drag tube heap (more than 800 m), we can see that most of the memory is occupied by the drag tube heap. We can also see that the previous plot is different. This time, there are not many large objects. We can get verification from the following command:

0: 011>! Dumpheap-min 85000
------------------------------
Heap 0
Address Mt size
473ac50c 000e18881400656 free
4750491c 000e1888916416 free
12881370 000e1888 1019752 free
1297a718 7912254c 118160
129974a8 000e1888 411144 free
129fbab0 79122610 121248
12a19460 79122610 4672296
12e8df88 79122610 4672296
13303ce0 000e1888453440 free
13373820 000e1888 449360 free
133e29d0 000e18882834040 free
13696a58 000e18883287480 free
139b9620 000e18884480064 free
13dff470 000e18883294072 free
141239f8 000e18883400864 free
Fc100038 000e1888513424 free
Fc17d9f8 000e1888906888 free
1fd5b080 7912254c 124488
1fd796c8 000e1888 966872 free
Fc500038 000e18888055264 free
5176ea18 7912254c 124488
Total 21 objects
------------------------------
Heap 1
Address Mt size
6e317958 000e18881976064 free
14860048 000e1888 113384 free
1487bb30 7912254c 124488
Total 3 objects
------------------------------
Heap 2
Address Mt size
Total 0 objects
------------------------------
Heap 3
Address Mt size
18860048 000e1888 2720656 free
18af83d8 7912273c 113368
Total 2 objects
------------------------------
Total 26 objects
Statistics:
Mt count totalsize Class Name
7912273c 1 113368 system. byte []
7912254c 4 491624 system. object []
79122610 3 9465840 system. Collections. hashtable + bucket []
000e1888 18 37199840 free
Total 26 objects

There are not many large objects (larger than 85000 bytes). There are 26 objects in total, and 18 are released. It seems that we cannot find the desired result from a large object. Then let's look at the objects with a lower level. What are the objects with a lower limit of 8500?

0: 011>! Dumpheap-min 8500
------------------------------
Heap 0
Address Mt size
790da154 790f9244 9280
0287c318 790f9244 16404
028804d4 790f9244 16404
02884f18 790f9244 32788
02aec324 7912273c 9604
02b1dffc 790f9244 16404
02b22be8 790f9244 32788
03038134 79122610 13248
0323006c 7912273c 11616
0586a068 7912254c 17904
3090bd60 7912273c 14836
30 adaffc 7912273c 54188
31fc6d04 7912273c 25728
32954478 79122610 28008
33f4296c 790f9244 16404
33f47558 790f9244 32788
44581a38 790f9244 16404
44586624 790f9244 32788
4461668c 79122610 28008
44dad90c 790f9244 16404
44db1ac8 790f9244 16404
44db1_c 790f9244 32788
44dd90f8 790f9244 16404
44dddce4 790f9244 32788
44df4004 790f9244 16404
44df81c0 790f9244 16404

......

3f028880 790f9244 16404
3f02d46c 790f9244 32788
3f182a1c 790f9244 16404
3f187608 790f9244 32788
3f1fd1dc 790f9244 16404
3f201dc8 790f9244 32788
3f25cef8 790f9244 16404
3f261ae4 790f9244 32788
3f2f36c8 790f9244 16404
3f2f82b4 790f9244 32788
3f7a87a8 7912254c 28204
4001eca4 000e18888760 free
401d399c 790f9244 21988
18860048 000e1888 2720656 free
18af83d8 7912273c 113368
Total 111 objects
------------------------------
Total 400 objects
Statistics:
Mt count totalsize Class Name
79122868 1 22676 system. int64 []
02803518 1 30908 system. Collections. Generic. dictionary '2 + entry [[system. String, mscorlib], [system. Web. configuration. authorizationsection, system. Web] []
7912273c 17 484412 system. byte []
7912254c 14 707912 system. object []
790f9244 330 8062572 system. String
79122610 14 9715704 system. Collections. hashtable + bucket []
000e1888 23 37249276 free
Total 400 objects

Okay. We found 400 objects in total. Now we can see a good phenomenon, that is, the object size is regular, and it is good to act normally. I want to see what these objects are?

0: 011>! Dumpheap-stat
------------------------------
Heap 0
Total 3617074 objects
------------------------------
Heap 1
Total 4237968 objects
------------------------------
Heap 2
Total 3868413 objects
------------------------------
Heap 3
Total 3860949 objects
------------------------------
Total 15584404 objects
Statistics:
Mt count totalsize Class Name
7ae7d438 1 12 system. Drawing. colorconverter
...

1cfb3180 33572 1074304 system. eventhandler 1 [[nbear. Web. Data. nbeardatasourceeventargs, nbear. Web. Data]
66408158 16788 1074432 system. Web. UI. htmlcontrols. htmllink
1c21a8b4 10418 1083472 shop. Code. Controls. initqscheckbox
663a2e58 5599 1097404 system. Web. httpresponse
1cb989d4 11192 1119200 Sof. Web. Controls. ajaxextender. namevalueautocomplete
1cfb2504 5596 1141584 ASP. default_master
1c21bff4 15627 1187652 shop. Code. Controls. initqstextbox
648f09f4 25240 1211520 system. configuration. configurationvalues
663e20a4 21658 1386112 system. Web. UI. htmlcontrols. htmlmeta
1c6c2074 3362 1425488 ASP. product_sortproductlist_aspx
663a6f84 5599 1433344 system. Web. Hosting. isapiworkerrequestinprocforiis6
6638e588 19015 1445140 system. Web. configuration. httphandleraction
648f1150 40166 1445976 system. configuration. configurationlockcollection
7a74db74 72771 1455420 system. componentmodel. eventhandlerlist + listentry
663b8af4 47082 1506624 system. Web. UI. rendermethod
663c8014 127279 1527348 system. Web. UI. compiledtemplatebuilder
1cb9b2c4 5595 1566600 Sof. Web. Controls. datacontrolpagerextender
791293b4 68360 1640640 system. Collections. Generic. List '1 [[system. String, mscorlib]
7a74da98 109651 1754416 system. componentmodel. eventhandlerlist
663b9534 47083 2071652 system. Web. UI. Control + controlrarefields
663b835c 32902 2105728 system. Web. UI. htmlcontrols. htmlgenericcontrol
648f0e74 106655 2133100 system. configuration. configurationvalue
1c6cc8ec 68160 2181120 nbear. Common. entity + queryproxyhandler
1c6cc7cc 68160 2181120 nbear. Common. entity + propertychangedhandler
6641160c 78330 2193240 system. Web. UI. webcontrols. listitem
663b631c 39171 2193576 system. Web. UI. webcontrols. contentplaceholder
7912273c 10349 2333704 system. byte []
663daa3c 127788 2555760 system. Web. UI. scriptkey
1d4295f4 80531 2576992 Lucene. net. Documents. Field
663b6f88 162409 2598544 system. Web. UI. attributecollection
7910c0f4 83555 2673760 system. eventhandler
6639b014 139791 2795820 system. Web. virtualpath
663bd1f8 78022 2808792 system. Web. UI. datasourceselectarguments
1d38c240 179183 2881432 ktdictseg. t_inner_pos []
1c1f2f44 27975 2909400 *. datasources. helpdatasources
653ba904 68162 3544424 system. Collections. Generic. dictionary '2 [[system. String, mscorlib], [system. Object, mscorlib]
1cf1_dc 30282 3633840 system. Web. UI. webcontrols. sofrepeater
790fdb60 336963 4043556 system. int32
6640f3ec 127279 4072928 system. Web. UI. buildtemplatemethod
79122414 40873 4332792 system. int32 []
663ca710 42532 4423328 system. Web. UI. webcontrols. Repeater
66410058 327377 5238032 system. Web. UI. statebag
790fd8b4 94095 5269320 system. Collections. hashtable
663dabb0 269441 6466584 system. Web. httpservervarscollectionentry
02805218 136320 7088640 system. collections. generic. dictionary '2 [[system. string, mscorlib], [system. collections. generic. list '1 [[system. object, mscorlib], mscorlib]
7910234c 312379 7497096 system. Collections. arraylist
7a7603a4 475247 7603952 system. Collections. Specialized. nameobjectcollectionbase + nameobjectentry
1d38c17c 308770 8645560 ftalgorithm. t_dfaunit
7a752890 470437 9408740 system. Collections. Specialized. hybriddictionary
663c4b3c 629654 10074464 system. Web. UI. stateitem
7a752968 360636 10097808 system. Collections. Specialized. listdictionary
1bd4e58c 63806 13016424 *. Entities. simpleproduct
66412704 304939 18296340 system. Web. UI. literalcontrol
7a752a34 916368 18327360 system. Collections. Specialized. listdictionary + dictionarynode
663b70f0 515385 18553860 system. Web. UI. controlcollection
663c00b8 285831 19436508 system. Web. UI. databoundliteralcontrol
663ed42c 307365 20900820 system. Web. UI. webcontrols. repeateritem
79122610 94101 32575176 system. Collections. hashtable + bucket []
000e1888 130 37305432 free
6641cf70 922079 40571476 system. Web. UI. Control + occasionalfields
7912254c 1433851 69899636 system. object []
790f9244 3565568 348747316 system. String
Total 15584404 objects
Fragmented blocks larger than 0.5 Mb:
ADDR size followed
473ac50c 1.3 MB 4750245c Lucene. net. Search. termscorer
4750491c 0.9 MB 475e44dc system. byte []
6e317958 1.9 MB 6e4fa058 system. byte []

Strange? How are all objects in the system. Web. UI space? Is the page not promptly released after being requested? As a result, a large number of page instances are stored in the drag-and-tube stack. From! The objects obtained by the dumpheap-min 8500 command select a meaningful object to see what is inside:

0: 011>! Gcroot 0x3ef16668
Note: roots found On Stacks may be false positives. Run "! Help gcroot"
More info.
Scan thread 9 osthread C8
Scan thread 17 osthread 354
Scan thread 18 osthread 12f8
Scan thread 19 osthread 14d4
Scan thread 20 osthread 10d4
Scan thread 22 osthread F58
Scan thread 10 osthread d2c
Scan thread 11 osthread 1180
ESP: 277eef4: Root: 06942b80 (*. serviceimpls. searchimpls. informationsearchservice)->
6e31491c (system. eventhandler '1 [[*. serviceinterfaces. searchedeventargs, *. serviceinterfaces])->
39f76528 (system. object [])->
4614dc40 (system. eventhandler '1 [[*. serviceinterfaces. searchedeventargs, *. serviceinterfaces])->
4614db74 (*. datasources. informationsearchdataview)->
3ef5d91c (*. datasources. informationsearchdatasource)->
3ef57f38 (Asp. product_sortproductlist_aspx)->
3ef02d08 (system. Web. httpcontext)->
3ef35a60 (system. Web. cachedpathdata)->
3ef57d08 (system. Web. configuration. handlermappingmemo)->
3ef43e10 (system. Web. configuration. httphandleraction)->
3ef1fde0 (system. configuration. runtimeconfigurationrecord)->
3ef0e920 (system. configuration. runtimeconfigurationrecord)->
3ef0ef18 (system. Collections. hashtable)->
3ef15af8 (system. Collections. hashtable + bucket [])->
3ef1e6bc (system. configuration. sectionrecord)->
3ef1e6e4 (system. configuration. sectioninput)->
3ef1e67c (system. configuration. sectionxmlinfo)->
3ef16668 (system. String)

It is still a page instance, but the root of the reference chain is on the instance of the searchimpls. informationsearchservice class. That is to say, as long as the instance cannot be released, the referenced informationsearchdatasource cannot be released because informationsearchdatasource itself has a page reference, page is referenced by other UI controls and context objects. As a result, the object of the entire page instance cannot be released? So what makes such a reference chain unable to be separated?

The breakthrough lies in 6e31491c (system. eventhandler '1 [[*. serviceinterfaces. searchedeventargs, *. serviceinterfaces])->. Is it an event proxy? So why is it so powerful? To clearly explain this problem, I have to explain the structure of my class.

Here I use nbear. IOC to design a search interface named isearchservice. This interface contains onsearching and onsearched events. This function is used to mount and search for previous and post-search event proxies. In datasource, this interface service is used as follows:

Nbear. IOC. servicefactory. getservice <isearchservice> (). onsearched + = new eventhandler (searched );

Searched is an instance function in datasource. In nbear. IOC, the default single instance mode is used for components. I have explained it here. Do you know the problem?

Asp.net instantiates the page for each user's access request. A page may be created with many instances. Different page instances will create different datasource instances, and different datasource instances will mount the searched method of their own instances to the isearchservice interface service. Because nbear. IOC uses a single instance for components. That is to say, as time advances, the searched event of the isearchservice interface service will be linked to the searched proxy of many different datasource instances. If the isearchservice interface service is not released, it means that these event proxies will not become inaccessible objects, and the chain of this proxy will always be connected, this directly affects the release of the datasource instance and the release of the page instance.

OK. How does it feel like it's in a circle? If you are still confused at this time, we suggest you understand the recycling mechanism of. net drag-and-drop objects. We recommend that you read the. NET Framework Program Design revision.

At this point, my analysis is basically over. The rest of the work is to verify the analysis results. Modified the program. After the upgrade. The starting price of the program is significantly reduced, and the unit of memory increase step size is also relatively small. After a period of stability, it will basically fluctuate over 260 MB. Everything looks so beautiful!

From this case, I summed up my own things. The design of a single instance, such as the single-piece instance mode. We always need to keep one point in mind, once a single instance is created. It will stay in memory with the process. All objects referenced by it will also exist and disappear like it. Before the application domain is uninstalled, GC cannot take them seriously!

In addition to the previous two articles, this is the "brilliant" result of windbg's first test in the past two days. I am surprised that I have not yet understood many concepts, and I do not know what commands are available. I can also use windbg to solve my actual problems. So through these three blogs, I hope to inspire a newbie like me. We can also make our own mistakes. However, this is just the beginning. If windbg is as simple as I said, then Microsoft will not have to have such a team. Haha

Links of the first two articles:

Windbg initial experience 2: Catch the villain

What should I do if the problem arises during the initial windbg experience?

A bu