Node. js and server load balancer under high concurrency

The emerging Node. js has attracted a lot of developers, providing us with a platform to quickly build high-performance network applications. I also started to step into node. js.

The emerging Node. js has attracted a lot of developers, providing us with a platform to quickly build high-performance network applications. I also began to gradually invest in node. in the embrace of js, I encountered some problems and some experience in the process of learning and using. I think it is necessary to write it out and share it as a summary.

As we all know, node. js is based on the v8 engine, so it does not support multithreading (Multi-threaded Module). to make full use of the server's Multi-core, you must use the Multi-process method. Load balancing between processes is a key issue.

Multi-process shared listening socket

Node. js and process-related modules include process, child_process, cluster, here, cluster is used to easily create The multi-process mode of The shared port (The cluster module allows you to easily create a network of processes that all share server ports ), this mode allows multiple processes to share a socket in the listening state, and the system assigns the accept connection to different sub-processes, which is also very simple to implement, cluster has done most of the work for you. here is a test case:

1 var cluster = require ('cluster ');
2 var http = require ('http ');
3 var numCPUs = require ('OS'). cpus (). length;
4
5 if (cluster. isMaster ){
6 // Fork workers.
7 for (var I = 0; I <numCPUs; I ++ ){
8 cluster. fork ();
9}
10
11 cluster. on ('eg', function (worker, code, signal ){
12 console. log ('worker' + worker. process. pid + 'died ');
13 });
14} else {
15 // Workers can share any TCP connection
16 // In this case its a HTTP server
17 http. createServer (function (req, res ){
18 res. writeHead (200 );
19 res. end ("hello world \ n ");
20}). listen (8000 );
21}

However, this completely dependent system load balancing has an important defect: on linux and Solaris, as long as the accept queue of a sub-process is null (usually the created sub-process), the system will allocate multiple connections to the same sub-process, this will cause extremely unbalanced load between processes. Especially when a persistent connection is used, the new coming connection per unit time is not high, and the accept queue of the sub-process is usually empty, which will cause the connection to be allocated to the same process continuously. Therefore, this load balancing relies entirely on the degree of idleness of the accept queue, and can achieve load balancing only when short connections are used and concurrency is very high, however, the system load will be very high and the system will become unstable.

How does Nginx work?

If you know about nginx, you may first think of how to use nginx,Nginx has one master and multiple worker processes. The master process listens to the port and is responsible for accept connection. it sends the socket of accept to each worker process, which receives and processes data.In linux, nginx uses socketpair to establish communication between master and worker processes, and uses APIs such as sendmsg and recvmsg to transmit commands and file descriptors. Does node. js support this scheme?

The answer is yes. the basis for this answer is that the child_process and cluster modules of node. js have a send method: child. send (message, [sendHandle]).

The second parameter of this method is the socket we want to pass, and the node. js document also provides a test case:

1 var normal = require ('child _ process'). fork ('child. js', ['normal']);
2 var special = require ('child _ process'). fork ('child. js', ['special ']);
3 // Open up the server and send sockets to child
4 var server = require ('net'). createServer ();
5 server. on ('connection', function (socket ){
6 // if this is a VIP
7 if (socket. remoteAddress = '74. 125.127.100 '){
8 special. send ('socket ', socket );
9 return;
10}
11 // just the usual dudes
12 normal. send ('socket ', socket );
13 });
14. server. listen (1337 );
Child. js

1 process. on ('message', function (m, socket ){
2 if (m = 'socket '){
3 socket. end ('you were handled as a' + process. argv [2] + 'person ');
4}
5 });
Simple and refined! It seems to be a perfect solution. Let's make it a normal http server:

Master. js

1 var http = require ('http '),
2 numCPUs = require ('OS'). cpus (). length;
3 cp = require ('child _ process '),
4 net = require ('net ');
5 var workers = [];
6 for (var I = 0; I <numCPUs; I ++ ){
7 workers. push (cp. fork ('app. js', ['normal']);
8}
9
10 net. createServer (function (s ){
11 s. pause ();
12 var worker = worker. shift ();
13 worker. send ('C', s );
14 workers. push (worker );
15}). listen (80 );
1 var http = require ('http '),
2 cp = require ('child _ process '),
3 net = require ('net ');
4 var server = http. createServer (function (req, res ){
5 res. writeHead (200, {"Content-Type": "text/plain", "Connection": "close "});
6 res. end ("hello, world ");
7 });
8 console. log ("webServer started on" + process. pid );
9 process. on ("message", function (msg, socket ){
10 process. nextTick (function (){
11 if (msg = 'C' & socket ){
12 socket. readable = socket. writable = true;
13 socket. resume ();
14 server. connections ++;
15 socket. server = server;
16 server. emit ("connection", socket );
17 socket. emit ("connect ");
18}
19 });
20 });
21
We have created an http server in the worker process, but this http server does not listen or bind a port. when receiving the socket transmitted by the master, we call the server. emit ("connection", socket); to trigger the connection event of the server. It looks good. it works normally after a simple test. this solution is almost perfect. After experiencing the failure of the shared listening socket solution, we migrated the service to this architecture.

However, we encountered a problem. We found that the cpu and memory of the master process gradually increased and eventually reached 100%, or node. js crash (Assertion 'fd _ to_send> = 0' failed), which made me crazy. we turned to node in a helpless way. js developers reported our problems on github (Issue #4587 ). That night, node. indutny, a js developer, found the problem, mainly because the main process was not destroyed after the socket was sent to the sub-process, but kept in the socketList, this will cause the socket to accumulate gradually in the main process and eventually reach the upper limit.

Indutny quickly solved this problem and submitted the commit the next day. According to this commit, indutny added the third optional parameter to the send function. the modified send function will be changed:

Child. send (message, [socket], [{track: false, process: false}])

Our master process does not need to track the status of each socket, so we can set it to false. At this point, this problem has been solved perfectly, and we hope this improvement can be released along with the next version of node. js.