Nodejs memory leak problem detailed _node.js

A previous chance found that react when the server renders, when node_env!= production, it can cause a memory leak. Concrete issues:https://github.com/facebook/react/issues/7406. With the node,react isomorphism and other technology widely used, node-side memory leakage and other problems should arouse our attention. Why node is prone to memory leaks and how to troubleshoot them after they appear, here is a simple introduction and examples to illustrate.

First, node is based on the V8 engine, and its memory management approach is consistent with V8. The following is a brief introduction to V8 's related memory effects.

V8 Memory Limits

Node based on V8 construction, through the V8 way to assign and manage JS objects. V8 the use of memory is limited (the 64-bit memory system under the 64-bit system is about 0.7G, the new generation of memory under the 32MB,32 system is about 16MB). Under such a restriction, a large memory object cannot be manipulated. If you accidentally touch this boundary, it will cause the process to exit.

Cause: V8 blocks JavaScript application logic when performing garbage collection, until garbage collection finishes and then executes the JavaScript application logic, which is called "Full pause" (Stop-the-world). If V8 heap memory for 1.5GB,V8 do a small garbage collection requires more than 50ms, do a non-incremental garbage collection even more than 1 seconds.

The default memory limit is cracked by node--max-old-space-size=xxx (in MB), node--max-new-space-size=xxx (in kilobytes), and the generation memory and the aged memory.

The heap composition of the V8

The V8 heap is not only composed of the two parts of the new generation and the Cenozoic, but can divide the heap into several different regions:

1. Cenozoic Memory Area: Most objects are allocated here, the area is small but the garbage back is particularly frequent.
2. The new generation pointer area: belong to the aged generation, here contains most of the possible objects pointing to other objects, most of the objects from the new generation will be moved here
3. Old generation data area: belong to the old generation, here only save the original data objects, these objects do not point to other objects pointers
4. Large object area: This holds objects that are larger than the size of other areas, each with its own memory, and garbage collection does not move large objects
5. Code area: The Code object, which is the object that contains the JIT instruction, is assigned here. The only memory area that has execute permissions
6. Cell area, attribute cell area, map area: storage cell, property cell and map, each region is the same size element, the structure is simple
   

GC Recycle type

An incremental GC

Indicates whether the garbage collector collects (increases) garbage while scanning the memory space and empties the garbage at the end of the scan cycle.

Non-incremental GC

When using a non-incremental garbage collector, a collection of garbage is about to be emptied.

The garbage collector only makes garbage collection for the new generation memory area, the aged pointer area and the aged generation data area. The object first enters the new generation memory with less space occupied. Most objects are quickly invalidated, and the non incremental GC directly reclaims these small amounts of memory. If some objects can not be recycled for a period of time, then go into the generation of memory area. This area performs infrequent incremental GC and takes a long time.

When will that lead to a memory leak?

Ways to leak memory

1. Memory leaks
2. Cache
3. Queue consumption is not timely
4. Scope not released

The memory composition of node is mainly the portion that is allocated through V8 and the portion that node allocates itself. The V8 garbage collection is limited primarily by V8 heap memory. The main cause of memory leakage: 1, cache, 2, queue consumption is not timely; 3, the scope is not released

Memory leak analysis

View V8 memory usage (in bytes)

Process.memoryusage (); 
  {
    ress:47038464,  
    heaptotal:34264656,  
    heapused:2052866  
  }

Ress: The resident memory portion of a process

HEAPTOTAL,HEAPUSED:V8 Heap Memory Information

View system memory usage (in bytes)

os.totalmem()
os.freemem()


Returns total system memory and idle memory

View garbage collection logs

Node--trace_gc-e "var a = []; for (var i = 0; i < 1000000; i++) {A.push (new Array);} ">> Gc.log//Output garbage collection log

Node--prof//Output node Run-time performance Log. Use Windows-tick.processor to view.

Analysis and monitoring tools

V8-profiler capture snapshots of V8 heap memory and analysis of CPU
Node-heapdump capture snapshots of V8 heap memory
Node-mtrace Analysis Stack Usage
Node-memwatch Monitoring of garbage collection

Node-memwatch

Memwatch.on (' Stats ', function (info) {
  Console.log (info)
})
Memwatch.on (' Leak ', function (info) {
  Console.log (info)
})

Stats event: A stats event is triggered each time a full heap garbage collection occurs. This event will pass the memory statistics.

{
"num_full_gc": 17,//The first several times full stack garbage collection
"NUM_INC_GC": 8,/  /incremental garbage collection
"Heap_compactions": 8,// The first several times to the generation of the
"Estimated_base": 2592568,//estimated base
"current_base": 2592568,//Current base
"min": 2499912,//min
"Max": 2592568,//Max 
"Usage_trend": 0//Usage trend
  }

Observation of NUM_FULL_GC and NUM_INC_GC reflect garbage collection.

Leak event: If the memory is still not released after 5 successive garbage collections, it means that the memory leak occurs. A leak event is triggered at this time.

{START:FRI, June 14:12:13 GMT,
End:fri, June 14:12:33 GMT,
growth:67984,
reason: ' Heap growt H over 5 consecutive GCs (20s)-11.67 mb/hr '
}

Heap diffing heap memory to troubleshoot memory overflow code.
Below, we demonstrate how to troubleshoot a location memory leak by using an example:

First we create an example that causes a memory leak:

App.js
var app = require (' Express ') ();
var http = require (' http '). Server (app);
var heapdump = require (' Heapdump ');

var leakobjs = [];
function Leakclass () {
  this.x = 1;
}

App.get ('/', function (req, res) {
  console.log (' Get/');
  for (var i = 0; i < 1000 i++) {
    Leakobjs.push (new Leakclass ());
  }
  Res.send ('  
 

Here we simulate a memory leak by setting up an array that is constantly increasing and not being recycled.

By using the Heap-dump module to record memory snapshots regularly, and through the Chrome Developer tool profiles to import snapshots, compare and analyze.

We can see that the size of the snapshot has been growing after the browser has accessed localhost:3000 and has been refreshed several times, and there has been a leak, even if no request has been made.

We then passed the Chrome Developer tool profiles to import the snapshots. By setting the comparison, compare the initial snapshot, send the request, smooth, and then send a memory snapshot of the 3-phase request. You can see that the Leakclass in the right new is always increasing. The delta is always positive, and the description is not reclaimed.

Summary

For memory leaks, you can use implantable memwatch, or periodically escalate process.memoryusage memory usage to monitor, and set alarm thresholds for monitoring.

When a memory leak problem is found, if allowed, you can run node-heapdump locally and use a timed build memory snapshot. And take the snapshot through the Chrome profiles to analyze the cause of the leak. If you cannot debug locally, use the V8-profiler output memory snapshot on the test server to compare and analyze JSON (requiring code intrusion).

You need to consider under what circumstances to open memwatch/heapdump. Consider the frequency of heapdump to avoid running out of CPU. You can also consider other ways to detect memory growth, such as direct monitoring of process.memoryusage ().

Beware of miscalculation, and the short peak of memory usage behaves much like a memory leak. If your app suddenly takes up a lot of CPU and memory, processing time may span several garbage collection cycles, so Memwatch is likely to misjudge it as a memory leak. However, in this case, once your app has finished using these resources, memory consumption will fall back to normal levels. Therefore, it is necessary to pay attention to the persistent reported memory leak, and can ignore one or two bursts of alarm.