Linux out of memory analysis (OOM)

Original article: http://blog.csdn.net/guomsh/article/details/6541015linuxhas a special :oom killer, a protection mechanism to avoid serious problems when memory is insufficient. This mechanism takes precedence over irrelevant processes, that is, when memory is seriously insufficient, to continue running, the kernel will select a process and kill it to release the memory to alleviate the memory insufficiency. However, this protection is limited and cannot completely protect the running of the process. In many cases, it is often seen that the process is still killed by Oom-killer when there is still memory available. The following information is found in the/var/log/messages log file: out of memory: killed process [pid] [process name]. this problem occurs because the kernel uses low memory to track all memory allocations. When low memory is exhausted, Oom-killer will kill the process no matter how much memory is left in high memory to keep the system running normally. The addressing range of a 32-bit CPU is limited. The Linux kernel defines the following three areas:

# DMA: 0x00000000-0x00999999 (0-16 MB)) # Lowmem: 0x01000000-0x037999999 (16-896 MB))-Size: 880 MB # highmem: 0x038000000-Lowmem (also called normal zone) is 880 MB in total and cannot be changed unless the hugemem kernel is used). For high-load systems, the OOM killer mechanism may be triggered due to poor lowmem usage. Memory allocation is a continuous region. If many fragments or lowfree fragments exist in lowmem and cannot be allocated to a contiguous memory area, oom killer is triggered. View the current lowfree value: CAT/proc/meminfo | grep lowfree view lowmem memory fragment: The command above CAT/proc/buddyinfo needs to be effective in the 2.6 kernel. There are the following methods to solve this problem: 1. upgrade to a 64-bit system. This is the best method, because all the memory belongs to low memory. If the prompt "out of memory" is displayed, it is really low memory used up, it is really oom. 2. If a 32-bit system is required, you can use the hugemem kernel. In this case, the kernel splits low/High memory in different ways, in most cases, sufficient ing between low memory and high memory is provided. In this case, you can install the hugemem kernel package and restart it. 3. If the hugemem kernel cannot be used, we can try to set the/proc/sys/Vm/lower_zone_protection value to 250 or greater. You can run the following command to view and set the value: CAT/proc/sys/Vm/lower_zone_protection echo 250>/proc/sys/Vm/lower_zone_protection or you can modify/etc/sysctl. CONF file to take effect after restart. The added content is as follows: VM. lower_zone_protection = 250 4. There is no way to do this. We should turn off Oom-killer. However, this option may cause the system to be suspended, so it depends on the actual situation. View the current Oom-killer status: CAT/proc/sys/Vm/Oom-kill close/enable Oom-killer: echo "0">/proc/sys/Vm/Oom-kill Echo "1">/proc/sys/Vm/Oom-kill or directly add it to/etc/sysctl. CONF file with the following content: VM. oom-kill = 0 at this time, when the process is killed by Oom-killer but fails to succeed, the relevant information is recorded in the/var/log/messages file. The information is as follows: "wocould have Oom-killed but/proc/sys/Vm/Oom-kill is disabled" 5. Disable Oom-killer and process it only for a single process, to protect a process, run the following command: ECHO-17>/proc/[pid]/oom_adj/proc/[pid]/OO In m_adj, the value range of oom_adj is-17 ~ 15. When the value is-17, the system will not kill the process with the specified PID, and-16 ~ 15 then, the/proc/[pid]/oom_adj value of the process increases exponentially (K * 2 ^ N), that is, the probability of the processes being killed increases exponentially. No matter how many processes the init (process number is 1) value is set, it will not be killed. 6, reference http://www.haw-haw.org/node/626 http://guogoul.com/2010/04/28/oom-killer-swap/ http://zhwen.org/xlog? P = 430 http://www.dbanotes.net/database/linux_outofmemory_oom_killer.html http://www.sealinger.com/archives/185 above is found from the network, combined with their own problems to supplement: at first because the system configuration is 2g, and no swap partition, therefore, the out of memory is generated every day, and the physical memory is increased and the swap partition is performed, which improves the situation. However, the out of memory situation still occurs after 2-3 days of running, later, I analyzed the log file messages and found the bold part. The analysis was caused by insufficient low memory. Later I tried to use sync; Echo 3>/proc/sys/Vm/drop_caches to release the memory, check whether/proc/buddyinfo memory is released. Jun 10 13:33:11 XX user. Warn kernel: DMA free: 3548kb min: 68kb low: 84kb high: 100kb active: 0kb inactive: 0kb present: 16384kb pages_scanned: 0 all_unreclaimable? Yesjun 10 13:33:11 XX user. warn kernel: lowmem_reserve []: 0 0 880 4080jun 10 13:33:11 XX user. warn kernel: dma32 free: 0kb min: 0kb low: 0kb high: 0kb active: 0kb inactive: 0kb present: 0kb pages_scanned: 0 all_unreclaimable? Nojun 10 13:33:11 XX user. Warn kernel: lowmem_reserve []: 0 0 880 4080
Jun 10 13:33:11 XX user. Warn kernel: normal free: 1340kb min: 3756kb low: 4692kb high: 5632kb active: 0kb inactive: 28kb present: 901120kb pages_scanned: 2456536 all_unreclaimable? YesJun 10 13:33:11 XX user. warn kernel: lowmem_reserve []: 0 0 25600jun 10 13:33:11 XX user. warn kernel: highmem free: 1615600kb min: 512kb low: 3928kb high: 7344kb active: 274366kb inactive: 24928kb present: 3276800kb pages_scanned: 0 all_unreclaimable? Nojun 10 13:33:11 XX user. warn kernel: lowmem_reserve []: 0 0 0 0jun 10 13:33:11 XX user. warn kernel: DMA: 1 * 4kb 1 * 8kb 1 * 16kb 0 * 32kb 1 * 64kb 1 * 128kb 1 * 256kb 0 * 512kb 1 * 1024kb 1 * 2048kb 0 * 4096kb = 3548 kbjun 10 13:33:11 XX user. warn kernel: dma32: emptyj
UN 10 13:33:11 XX user. warn kernel: Normal: 1 * 4kb 1 * 8kb 1 * 16kb 1 * 32kb 0 * 64kb 0 * 128kb 1 * 256kb 0 * 512kb 1 * 1024kb 0 * 2048kb 0 * 4096kb = 1340kbJun 10 13:33:11 chinasoft user. warn kernel: highmem: 8192 * 4kb 7506 * 8kb 5166 * 16kb 2932 * 32kb 1528 * 64kb 688 * 128kb 253 * 256kb 86 * 512kb 67 * 1024kb 12 * 2048kb 234 * 4096kb = 1615600kb
 
The above are problems you have encountered, and there must be something wrong with the handling. please correct me if you find it! Supplement: the kernel is 2.26.16.