Target
(1) understand the impact of erlang concurrent process scheduling on the load of each cpu core in a multi-core cpu environment;
(2) memory increase mechanism of Erlang virtual machine;
(3) scheduling of Erlang processes;
(4) monitor cpu usage in Linux
Lab Environment
Lenovo minicomputer: Operating System: RedHat Enterprise LinuxServer release6.4 (Santiago)
Kernel version: Linux server1 2.6.32-358. el6.x86 _ 64 #1 SMP
CPU model: Intel (R) Xeon (R) CPU E7-4820 @ 2.00 Ghz;
4 CPUs, each with 8 Physical cores, 32 physical cores, and 64 logical Cores
Memory: 125 GB
Disk: 289 GB
Erlang OTP: Erlang/OTP 17 [erts-6.0] [source] [64bit] [smp: 64: 64] [async-thread: 10]
Testing Erlang concurrent processes
Test code
-Module (test ).
-Export ([start_proc/1]).
Start_proc (Num)->
Case Num =: = 0
True-> OK;
False-> spawn (fun ()-> loop () end), start_proc (Num-1)
End.
Loop ()-> loop ().
Cpu running status
Test: start_proc (1000000) after 1000000 processes are started, run mpstat-PAll 2 10
Concurrent I/O
Test code
-Module (testio ).
-Export ([start_proc/1]).
Name ()->
{A, B, C} = erlang: now (),
Integer_to_list (A) ++ Integer_to_list (B) ++ Integer_to_list (C ).
Io ()->
{OK, Fd} = file: open ("./data/" ++ name (), [write, raw, binary, append]),
File: write (Fd, "Hi zcc, nice to meet you "),
File: read (Fd, 23 ),
File: close (Fd ).
Start_proc (Num)->
IO = fun ()-> io () end,
Case Num =: = 0of
True-> OK;
False-> spawn (IO), timer: sleep (1), start_proc (Num-1)
End.
View kernel information in Linux
Vmstat-n 3 refreshes every 3 seconds
Analysis of cpu utilization using sar-u 2 10
Analyze the length of the running process queue sar-q 2 10
Cpu % usr % nice % sys % iowait % irq % soft % steal % guest % idle
Mpstat-P 1: View cpu information on instance 1
% Steal: Percentage of idle waiting time of virtual cpu
% Nice: Percentage of cpu time in user mode with nice Value
Top-M, press the F key, press j, and press enter to see which core the process is running on.
Conclusion
(1) by default, Erlang's smp scheduling process schedualer should check the cpu, rather than the number of cpus. This environment has four CPUs, 32 physical cores, 64 logic cores after hyper-threading is enabled, and 64 scheduling processes;
(2) When erlang is concurrent with multiple processes, it can be seen from the cpu load that each process will run evenly on each core, instead of the excessive load on a certain core, when a core load is too small, this task is done by an operating system, so programmers do not have to worry about it;
(3) the memory of the Erlang virtual machine will automatically increase from the host memory as the number of processes increases. Unlike the Java virtual machine, the memory will not automatically increase before jvm1.6, you can only manually increase the jvm memory. The memory of the physical host can be shared after 1.7. The Erlang virtual machine does not suppress the memory growth mechanism. The VM constantly allocates memory, forcing the system to use the swap zone until the VM memory is exhausted and becomes very slow. The 'private heap 'and queue-based programming models designed by the Erlang virtual machine are both advantages and disadvantages. when running the erlang virtual machine in the production environment, a system-level detection is required, in this way, processes can be killed when erlang memory usage soar.
(4) The Erlang scheduler runs on an OS thread. The OS determines whether the scheduler runs on a core. Generally, the OS ensures that the thread runs on a core during execution;
(5) estimate the maximum number of processes in a system = total memory/erlang: process_info (self (), memory), and obtain erlang wordsize = erlang: system_info (wordsize ). A process in Erlang occupies about 2667 bytes of memory.
(6) Test Results
Number of started processes |
Virtual Machine memory |
Start Time |
0.1 million |
299 M |
12.3 s |
1 million |
2.7 GB |
6.8 s |
10 million |
27.2 GB |
57.6 s |
55164851 |
132 GB |
400 s |