Linux OOM Auto Kill process

Problem Description:

Today after work, log in an intranet test server, found that some of the process is missing (Nginx/php-fpm/mysql/crond).

Workaround:

1, start these processes first, to ensure the normal provision of services.

2, check the server log information, troubleshooting.

Shell > Less/var/log/messages-20160905Sep3  to: to:TenLocaldomain Kernel:out of Memory:kill process3145(MYSQLD) Score -or Sacrifice Childsep3  to: to:TenLocaldomain kernel:killed Process3145Uid804, (mysqld) total-vm:9170936kb, ANON-RSS:246880KB,file-Rss:32kbsep3  to: +: ALocaldomain Kernel:out of Memory:kill process20889(Nginx) Score2or Sacrifice Childsep3  to: +: ALocaldomain kernel:killed Process20889Uid803, (Nginx) total-vm:130256kb, anon-rss:216kb,file-rss:12kb

3, found similar log information, online search information.

4, the data show that this is due to the system memory is not enough to trigger the Linux Kernel OOM (out of memory killer) protection mechanism, will take up a large-footprint process to kill, to ensure that the system runs properly.

Sep3  to:Panax Notoginseng:TenLocaldomain Kernel:mysqld invoked oom-killer:gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0Sep3  to:Panax Notoginseng:TenLocaldomain kernel:mysqld cpuset=/mems_allowed=0-1Sep3  to: the: -Localdomain Kernel:crond invoked oom-killer:gfp_mask=0x84d0, order=0, oom_adj=0, oom_score_adj=0Sep3  to: +: -Localdomain Kernel:crond cpuset=/mems_allowed=0-1Sep3  to: +: $Localdomain kernel:php invoked oom-killer:gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0Sep3  to: +:Panax NotoginsengLocaldomain kernel:php cpuset=/mems_allowed=0-1

5, also found a situation, memory enough, but still triggered the OOM kill process. This involves a low Memory knowledge point. The kernel uses low memory to keep track of all allocations.

Note: The low memory and high memory are distinguished only in 32-bit operating systems, and the low memories in 64-bit systems are all of the RAM space.

Shell > Free-LM Total used Freeshared buffers Cachedmem:11901       1039      10861          0          -        133Low :11901       1039      10861High :0          0          0-/+ Buffers/cache:858      11042Swap:8191          2       8189

# you can see: The low size is as large as the total memory, and the high is 0.

6. What should I do?

> Add memory, use 64-bit operating system.

> Reasonable allocation of memory space, such as the PHP Accelerator cache space, and so on, periodically restart the PHP-FPM process.

> Manually release memory: sync; Echo 3 >/proc/sys/vm/drop_caches

> uses the Hugemem kernel, which splits low/high memory in different ways, and in many cases provides enough low memory to high memory mappings.
Install the HUGEMEM kernel RPM package and restart the server.

> Specifies not to kill a process: echo-17 >/proc/$ (pidof mysqld)/oom_adj #-17 Disables oom for this process

> echo "vm.panic_on_oom = 1" >>/etc/sysctl.conf; Sysctl-p # Close OOM (after two or three kinds of caution)

Sync Echo 3 >/proc/sys/vm/drop_caches

Linux OOM Auto Kill process