Optimized solution for memory crunch in Erlang server

　　

Issue: Server 1 million people online, 16G memory is almost eaten up. Player process occupies high memory

Workaround:

The first step:
Erlang:system_info (Process_count). See if the number of processes is normal and exceeds the maximum number of processes for the Erlang virtual machine.
Step Two:
View the node's memory bottleneck location
> Erlang:memory ().
[{total,2099813400},
{processes,1985444264},
{processes_used,1985276128},
{system,114369136},
{atom,4479545},
{atom_used,4477777},
{binary,22756952},
{code,10486554},
{ets,47948808}]
The display memory is mostly consumed on the process, thus determining that the process is consuming a lot of memory

Step Three:
View the most memory-intensive processes

>spawn (Fun ()-Etop:start ([{Output, text}, {interval, 1}, {lines, +}, {sort, memory}]).
(Start etop in output text, with an interval of 1 seconds, an output line number of 20 rows, sorted in memory.) Here spawn a new process to output etop data without affecting Erlang shell input.)

Fourth step: View the state of the process that occupies the highest memory
>erlang:process_info (PID (0,12571,0)).
[{current_function,{mod_player,send_msg,2}},
{initial_call,{erlang,apply,2}},
{status,waiting},
{message_queue_len,0},
{messages,[]},
{links,[<0.12570.0>]},
{dictionary,[]},
{Trap_exit,false},
{Error_handler,error_handler},
{Priority,normal},
{Group_leader,<0.46.0>},
{total_heap_size,12538050},
{heap_size,12538050},
{stack_size,10122096},
{reductions,3795950},
{garbage_collection,[{min_bin_vheap_size,46368},
{min_heap_size,233},
{fullsweep_after,65535},
{minor_gcs,0}]},
{suspending,[]}]

where "{total_heap_size,12538050}," indicates a memory consumption of 12358050 words (32-bit system word size is 4, 64-bit system word size is 8, can be Erlang:system_info ( Wordsize), 64-bit system will be nearly 100M, too exaggerated!

Fifth Step:
Manual GC Recycle, hope the problem can be solved
> Erlang:garbage_collect (PID (0,12571,0)).
True
Look at the process memory again, find that there is no change!GC no back received any resources, so the memory is still playing a role, not recycled!

Sixth step:
Do not suspect the system, first of all to suspect their own code
Careful observation of the code, its approximate structure is as follows:
Send_msg (Socket, Pid)
Try
Receive
{Send, Bin}
...
{inet_reply, _sock, Result}
...
Catch
_:_->
Send_msg (Sock,pid)
End.
The purpose is to loop through the data and then send it, using the Try...catch to catch the exception.
This piece of code is not a tail recursion! Try...catch will save the appropriate information in the stack, the exception capture needs to be placed inside the function, so send_msg Last call is Try...catch, not itself, so not the tail recursion!
Can be verified by code:
Cat Test.erl
-module (Test).
-compile ([Export_all]).


T1 ()
Pid = Spawn (Fun (), do_t1 () end),
Send_msg (Pid, 100000).

T2 ()
Pid = Spawn (Fun (), do_t2 () end),
Send_msg (Pid, 100000).

Send_msg (_pid, 0)
Ok
Send_msg (Pid, N)
Pid!<<2: (N) >>
Timer:sleep (200),
Send_msg (Pid, N-1).

DO_T1 ()
Erlang:garbage_collect (self ()),
Result =erlang:process_info (self (), [Memory, Garbage_collection]),
Io:format ("~w~n", [Result]),
Io:format ("Backtrace:~w~n~n", [Erlang:process_display (Self (), backtrace)]),
Try
Receive
_->
DO_T1 ()
End
Catch
_:_
DO_T1 ()
End.

Do_t2 ()
Erlang:garbage_collect (self ()),
Result =erlang:process_info (self (), [Memory, Garbage_collection]),
Io:format ("~w~n", [Result]),
Io:format ("Backtrace:~w~n~n", [Erlang:process_display (Self (), backtrace)]),
Receive
_,
Do_t2 ()
End.

Version 1:erlctest.erl && erl-eval "test:t1 ()"
Version 2:erlctest.erl && erl-eval "test:t2 ()"
You'll see that the call stack for version 1 code is growing, memory is growing, and the version 2 function call address remains the same, and memory hasn't changed!

Summarize:
1, in the server programming, the cycle must ensure that the tail recursion;
2, use OTP as much as possible, and avoid this problem if you replace the handwritten loop with Gen_server.

Transferred from: http://www.2cto.com/os/201405/299927.html

Optimized solution for memory crunch in Erlang server