[Debug practice] windbg + Performance Monitor solves a memory leakage problem

[Debug practice] windbg + Performance Monitor solves a memory leakage problem

Solve a memory problem that has been plagued for many days this afternoon. Record the analysis process

Problem description:
W3wp memory increases slowly, and it will reach about 1.3g at last, so CPU resources are not very tight.

Analysis process:
Previous problems were caused by hosting memory leaks. Based on previous experience, dump a file, a 1.4g large file, and windbg opened the file,
0: 000>! Eeheap-GC
GC heap size 0x22cf6b80 (584018816)
Because a recently cached project has just been launched and some data is directly stored in the memory, more than 500 mb is an expected size. At least 1.4g, the main reason for this memory outbreak should not be this.
Yes! After reading the eeheap-loader, there is no typical module loading problem.
But looking at the contents in the heap is not always a bad thing.
0: 000>! Dumpheap-stat
001c91f8 1216 78189440 free
7912dae8 14755 142620572 system. byte []
790fd8c4 1625830 201229328 system. String
It is a common big string problem to find out more than KB:
! Dumpheap-MT 790fd8c4-min 100000
--- Omit long content ----
27f93108 790fd8c4 241560
Total 76 objects
------------------------------
Total 320 objects
Statistics:
Mt count totalsize Class Name
790fd8c4 320 54798168 system. String
I found several strings up to 1 MB, which are basically some order information strings returned by the database.
When it was depressing, the O & M Department provided the second dump file and analyzed the file using the same method above. The conclusion was that the GC heap size was similar, however, the number of dump files increased by more than 100 MB.
I began to suspect that I had encountered an unmanaged leak.

After several twists and turns, the O & M Department agreed that I could directly connect to the server to view the information. When I enable the Performance Monitor (perfmon), I first loaded private bytes, virtual bytes, and bytes in all heaps,
In a familiar scenario, private bytes and virtual bytes grow together, but bytes in all heaps does not fluctuate much. It is more like a non-hosted leak.
Therefore, debugdiag was used for monitoring, but unfortunately the growth process was slow, and a dump was captured when it increased by dozens of megabytes,
In the report generated by analysis:
Webengine. dllIs responsible13.49 MbytesWorth of outstanding allocations. The following are the top 2 memory consuming functions:

Webengine! Bufferpool: getbuffer + 4A:13.48 MbytesWorth of outstanding allocations.
Webengine! Memalloc + 22:15.92 KbytesWorth of outstanding allocations.
Iisutil. dllIs responsible1.97 MbytesWorth of outstanding allocations. The following are the top 2 memory consuming functions:

Iisutil! Alloc_cache_handler: alloc + 4f:1.43 MbytesWorth of outstanding allocations.
Iisutil! Buffer: reallocstorage + 2e:530.90 KbytesWorth of outstanding allocations.
The monitoring time is insufficient and there are not many clues,
An interesting image of the Performance Monitor captured in the subsequent wait process: As shown in, the dark brown line is the size of the large object (LOH) in the managed heap,
When it suddenly rises, both virtual bytes (fine purple red line) and private bytes (fine yellow line) increase, but after a while the size of the LOH comes down, virtual bytes does not decrease significantly
So in the next LOH increase, the two files are dumped at the high point and the low point respectively.

Use windbg to open two dump files. After reading the dumpheap content, there are major differences in strings,

Contents before LOH increase:
790fd8c4 503444 51605012 system. String
7912dae8 5168 61353852 system. byte []
Content After LOH increase:
7912dae8 5833 62697008 system. byte []
790fd8c4 558725 67602548 system. String

Haha, large objects should appear in system. String,
0: 000>! Dumpheap-MT 790fd8c4-min 1000000
Heap 1
Address Mt size
222a5210 790fd8c4 1738704
3cd30038 790fd8c4 17563672
Total 2 objects

There is a string up to 17 MB ,! Do display string: <string is invalid or too large to print>
View du directly

0: 000> du 3cd30038 3cd30038 + 1000
3cd30038 ". Large... with G <diffgr: DiffGram xmlns: MSD"
3cd30078 "ata =" urn: Schemas-Microsoft-com: X"
3cd300b8 "ML-msdata" xmlns: diffgr = "urn: Sch"
3cd300f8 "EMAS-Microsoft-com: XML-DiffGram -"
3cd30138 "V1">... <newdataset>... <Table"
3cd30178 "diffgr: Id =" Table1 "msdata: rowor"
3cd301b8 "Der =" 0 "diffgr: haschanges =" modif"
3cd301f8 "ied">... <reser_no> 16644349 <"
---- A bunch of content is omitted later

Then we foundCodeProbably becauseProgramThe member directly binds the result of an SQL query to the DataGrid,
The result set can contain tens of thousands of data records on the right.
The next step is to reproduce a big data-intensive business scenario with confidence. Finding an idle server will repeat the problem as long as one person refreshes the order query page. Now, the problem is basically located.

Let's look back at the dump captured by debugdiag. The generated file is shown as follows:
Mscorwks. dll(A known Windows Memory Manager) is responsible202.89 MbytesWorth of outstanding allocations. These allocations appear to have originated from the following module (s) and function (s ):

0x79e7495b

WinHTTP. dllIs responsible71.99 MbytesWorth of outstanding allocations. The following are the top 2 memory consuming functions: WinHTTP! Rockall_back_end: newarea + 9C:71.99 MbytesWorth of outstanding allocations.

Where

Function	Mscorwks! Eevirtualalloc + 104
Allocation type	Virtual Memory Allocation (s)
Allocation count	79 allocation (s)
Allocation size	199.56 Mbytes
Leak Probability	41%

64.00 Kbytes 32.00 Mbytes 3.06 Mbytes

72 allocation (s) 6 allocation (s) 1 Allocation (s)

Top 3 Allocation sizes by total size

32.00 Mbytes 64.00 Kbytes 3.06 Mbytes

192.00 Mbytes 4.50 Mbytes 3.06 Mbytes

Call Stack sample 1

Address 0x3cd30000
Allocation time 02:08:39 since tracking started
Allocation size 32.00 Mbytes
Function source destination
0x79e7495b Kernel32! Virtualalloc
System. Io. memorystream. set_capacity (int32) 0x79e756b0
System. Io. memorystream. ensurecapacity (int32)
System. Io. memorystream. writebyte (byte) system. Io. memorystream. ensurecapacity (int32)
System. Io. binarywriter. Write (byte)
System. Web. util. altserialization. writevaluetostream (system. Object, system. Io. binarywriter)
System. Web. sessionstate. sessionstateitemcollection. writevaluetostreamwithassert (system. Object, system. Io. binarywriter) system. Web. util. altserialization. writevaluetostream (system. Object, system. Io. binarywriter)
System. Web. sessionstate. sessionstateitemcollection. serialize (system. Io. binarywriter) system. Web. sessionstate. sessionstateitemcollection. writevaluetostreamwithassert (system. Object, system. Io. binarywriter)
System. Web. sessionstate. sessionstateutility. serialize (system. Web. sessionstate. sessionstatestoredata, system. Io. Stream) system. Web. sessionstate. sessionstateitemcollection. serialize (system. Io. binarywriter)
System. web. sessionstate. sessionstateutility. serializestoredata (system. web. sessionstate. sessionstatestoredata, int32, byte [] byref, int32 byref) system. web. sessionstate. sessionstateutility. serialize (system. web. sessionstate. sessionstatestoredata, system. io. stream)
System. web. sessionstate. sqlsessionstatestore. setandreleaseitemexclusive (system. web. httpcontext, system. string, system. web. sessionstate. sessionstatestoredata, system. object, Boolean) system. web. sessionstate. sessionstateutility. serializestoredata (system. web. sessionstate. sessionstatestoredata, int32, byte [] byref, int32 byref)
System. Web. sessionstate. sessionstatemodule. onreleasestate (system. Object, system. eventargs)
System. Web. sessionstate. httpsessionstate. set_item (system. String, system. Object) 0x2a8852a
System. Web. httpapplication + synceventexecutionstep. system. Web. httpapplication. iexecutionstep. Execute ()
System. Web. httpapplication. system. Web. ihttpasynchandler. beginprocessrequest (system. Web. httpcontext, system. asynccallback, system. Object)
0x79f047fd
Webengine! Httpcompletion: processrequestinmanagedcode + 1cb
Webengine! Httpcompletion: processrequestinmanagedcode + 1cb
Webengine! Httpcompletion: processcompletion + 48 webengine! Httpcompletion: processrequestinmanagedcode
Webengine! Corthreadpoolworkitemcallback + 1A
0x79f024cf
0x79f0202a
0x79f95a2e
Ntdll! Zwtestalert + 15
Ntdll! Ntcontinue + 12
Ntdll! Kiuserapcdispatcher + 3A NTDLL! Ntcontinue
Kernel32! Basethreadstart + 34