Nodejs the cause analysis of hbase and its relationship with JavaScript's garbage collection mechanism through thrift operation

Recently, using Nodejs to manipulate hbase through thrift, I wrote a script that constantly sends HTTP requests to get the data required under HBase, but during run the for loop is not executed completely and will get stuck after executing part Will never enter hbase under the data, the socket hang up error, check a lot of information is not resolved. At that time, it is the concurrency number problem of hbase, its concurrency limit leads to the limit of resource load, and then constantly test to find the reason, in fact, the ability of hbase to deal with the concurrency is irrelevant, the real reason is that jsvascript garbage collection mechanism makes resources use to reach the bottleneck, Here is the comparison before and after the code is processed:

1 varThrift = require (' Thrift ')),2HBase = require ('./gen-nodejs/hbase.js ')),3Hbasetypes = require ('./gen-nodejs/hbase_types.js ')),4Connection = thrift.createconnection (' localhost ', 9090, {5 Transport:thrift. Tframedtransport,6 Protocol:thrift. Tbinaryprotocol7     });8 9 varClient =thrift.createclient (hbase,connection);Ten  One /*router.get ('/', function (req, res) A { -  - res.render (' index ', {title: ' Welcome '}); the  - });*/ -  -Router.get ('/search ',function(req, res) + { -  +     varDatetimearray = {}; A     varValueArray = {}; at     varSearchplatebegin =NULL; -     varSearchplateend =NULL; -     varSearchdetailsbegin =NULL; -     varSearchdetailsend =NULL; -     varConvertreversearray =NewArray (); -}

1 /*router.get ('/', function (req, res)2 {3 4 res.render (' index ', {title: ' Welcome '});5 6 });*/7 8Router.get ('/search ',function(req, res)9 {Ten  One     varDatetimearray = {}; A     varValueArray = {}; -     varSearchplatebegin =NULL; -     varSearchplateend =NULL; the     varSearchdetailsbegin =NULL; -     varSearchdetailsend =NULL; -     varConvertreversearray =NewArray (); -  +      varThrift = require (' Thrift ')), -HBase = require ('./gen-nodejs/hbase.js ')), +Hbasetypes = require ('./gen-nodejs/hbase_types.js ')), AConnection = thrift.createconnection (' localhost ', 9090, { at Transport:thrift. Tframedtransport, - Protocol:thrift. Tbinaryprotocol -     }); -  - varClient =thrift.createclient (hbase,connection); -}

The only difference between the two pieces of code is whether the thrift connection HBase is placed in a few lines of code in the routing search, in the running script to constantly request the route that is the reason, because of constant request, every request will be once thrift connection HBase, that code seems simple, In fact, the operation behind the complex, including the addition of libraries in HBase, structure organization, memory allocation and so on, so that the loop to create a lot of thrift objects, because the JavaScript garbage collection mechanism of delay can not be recycled, This increase in the number of methods objects will cause thrift to request data blocking from HBase, as we have seen, stuck there.

Here's a look at the JavaScript recycling mechanism:

There are now two ways that big browsers typically use garbage collection: Tag purge, reference count.

1. Mark Clear:

This is the most common way of garbage collection in JavaScript, when variables enter the execution environment, such as declaring a variable in a function, the garbage collector marks it as "going into the environment," and when the variable leaves the environment (the function execution ends) marks it as "out of the environment." There are many ways to tag, such as the reversal of a particular bit, the maintenance of a list, and so on, which is not important, it is important to use what strategy, in principle, can not release the memory of the variables into the environment, they can be called at any time.

The garbage collector will tag all variables stored in memory at run time, then remove the variables in the environment and the variables referenced by the variables in the environment (closures), after which there are still tags to be deleted, because variables in the environment are inaccessible to these variables. The garbage collector then encounters the space occupied by these tagged variable machines.

2. Reference count:

Another less common garbage collection strategy is the reference count. The meaning of the reference count is the number of times each value is referenced by the tracking record. When a variable is declared and a reference type is assigned to the variable, the number of references to that value is 1. Conversely, if a variable that contains a reference to this value has another value, the number of references to this value is reduced by 1. When this number of references becomes 0 o'clock, it means that there is no way to access the value again, so that it can take up the memory space it occupies. In this way, the next time the garbage collector runs, it frees the memory of those values that have a reference count of 0.

For example, object A has a property that points to object B, and object B has a property that points to object A, which references each other.

function Test () {            var a={};             var b={};            A.prop=b;            B.prop=A;        }

So A and B references are 2, even after the test () execution completes, two objects have left the environment, under the policy of the tag purge is not a problem, leaving the environment is cleared, but not under the reference counting policy, because the two objects are still 2 references, will not become 0, So its footprint will not be cleaned up, if the function is called multiple times, so that there will be no more space to be recycled, resulting in memory leaks.

Reduce garbage collection in javascript:

1. Array Optimization:

assigning [] to an Array object is a shortcut to emptying the array (for example: arr = [];), but it is important to note that this creates a new empty object and turns the original array object into a small piece of memory garbage! In fact, assigning a value of array length to 0 (arr.length = 0) can also achieve the purpose of emptying the array, and it can also realize the reuse of the array and reduce the generation of memory garbage.

2. Function Optimization:

Methods are typically created at initialization time, and are rarely dynamically allocated at run time, making the method that leads to memory garbage generation not so easy to find. But from another point of view, it's easier for us to look for, because as long as it's a dynamic way of creating a method, it's possible to generate memory garbage.

setTimeout (    function(self) {                          returnfunction  () {              Self.tick ();    };}) (This), 16)

Every 16 milliseconds to call once This.tick (), well, at first glance seems to be no problem, but carefully pondering, each call has returned a new method object, which led to a lot of method object garbage!

You can save a method as a return value, for example:

this. Tickfunc = (    function(self) {      returnfunction () {                self.tick ();      };    }) (this); // In the tick () functionsetTimeout (this. Tickfunc, 16);

Instead of creating a new method object each time, the same method object is reused in each frame. The advantages of this approach are obvious, and the idea can be applied in any case where the method is a return value or a method is created at run time.

Nodejs the cause analysis of hbase and its relationship with JavaScript's garbage collection mechanism through thrift operation