Linux Terminal reports Out of memory: Kill process [PID] [process name] score problem analysis

The screen displays all Out of memory: Kill process [PID] [process name] score, although it is known that this is a linux self-protection for memory cleaning, however, I learned more details.

[Cause analysis]

Out of memory, which is usually caused by insufficient system Memory due to a large number of requests from applications at a certain time. This usually triggers Out of memory (OOM) killer in the Linux kernel, OOM killer will kill a process to free up memory for the system, instead of causing the system to crash immediately.

The Linux kernel allocates memory according to application requirements. Generally, the application allocates memory but not all of them are actually used. To improve performance, this part of useless memory can be reserved for use, this part of memory belongs to every process. It is troublesome to directly recycle and use the kernel. Therefore, the kernel uses an over-commit memory method) to indirectly use this part of "idle" memory to improve the overall memory usage efficiency. In general, this is no problem, but it is troublesome when most applications consume their own memory, because the memory requirements of these applications exceed the physical memory (including swap) the kernel (OOM killer) must kill some processes to free up space to ensure the normal operation of the system.

Some people may find that there is still a lot of memory left? Why is the error reported that the memory is insufficient? This is because of a 32-bit system. If Low-memory is used up, this problem may occur. So what is low-memory?

The kernel uses low memory to track all memory allocations. In this case, a 16 GB memory system consumes more low memory than a 4 GB memory system, which may be 4 times more. This extra pressure exists from the very moment you start the system, because the kernel structure must be adjusted to potentially track more than four times the memory allocation

OOM Killer is a protection mechanism used to prevent Linux from getting too serious problems when the memory is insufficient, and kill irrelevant processes.

Addressing is limited in a 32-bit CPU architecture. The Linux kernel defines three regions:

# DMA: 0x00000000-0x00999999 (0-16 MB)
# LowMem: 0x01000000-0x037999999 (16-896 MB)-size: 880 MB
# HighMem: 0x038000000-LowMem (also called normal zone) has a total of 880 MB and cannot be changed (unless the hugemem kernel is used ). For high-load systems, OOM Killer may be caused by poor LowMem utilization. One possible reason is that LowFree is too small, and the other reason is that the LowMem is full of fragments and the request cannot reach the continuous memory area.

In addition, in a 64-bit system, low-memory is the total memory space.

Check the statuses of low memory and high memory:

[Root @ localhost ~] # Free-lm
Total used free shared buffers cached
Mem: 32105 11305 20800 0 176 5402
Low: 32105 11305 20800
High: 0 0 0
-/+ Buffers/cache: 5726 26379
Swap: 32767 0 32767
[OOM killer principle]

Oom_killer rates each process and determines which process to kill based on the level of points. This points can be adjusted. root-authorized processes are generally considered important and should not be killed easily, therefore, you can get a 3% discount (-= 30; the lower the score, the less likely it is to be killed ). In the user space, we can operate on the kernel parameters of each process to determine which processes are not so easy to be killed by OOM killer. For example, if you do not want the MySQL process to be killed easily, you can find the MySQL running process number and adjust oom_score_adj to-15 (note that the smaller the points, the less likely it is to be killed)

Ps aux | grep mysqld
Mysql 2196 1.6 2.1 623800 44876? Ssl/usr/sbin/mysqld
 
Cat/proc/2196/oom_score_adj
0
Echo-15>/proc/2196/oom_score_adj

[Solution]

What should I do if I know the principle?

1. Increase the memory size.

Are we sure we want to increase the memory? Otherwise, how can we call it a server?

2. Upgrade to the 64-bit operating system.

The 64-bit operating system has no restrictions on low-memory.

3. Use the hugemem kernel.

This kernel separates low/high memory in different ways, and provides enough ing between low memory and high memory in most cases. In most cases, this is a simple solution: install the hugemem kernel RPM Package and restart it.

4. Configure OOM killer

Adjust OOM killer's behavior through some kernel parameters to prevent the system from continuously killing processes. For example, the kernel panic can be triggered immediately after OOM is triggered, and the system is automatically restarted 10 seconds later.

Echo "vm. panic_on_oom = 1">/etc/sysctl. conf
Echo "kernel. panic = 10">/etc/sysctl. conf
Sysctl-p

5. Disable/enable oom-killer (use with caution)

Echo "0">/proc/sys/vm/oom-kill
Echo "1">/proc/sys/vm/oom-kill