MySQL OOM System 2 OOM Killer, oomkiller

MySQL OOM System 2 OOM Killer, oomkiller

Here is a question. Who should Kill? Generally, I want to know a little about some Linux kernel users who use the most frequently and Kill them. This is of course the first important factor to consider for Linux kernel, but it is not exactly the same. Let's look up some Linux kernel information, you can know who Kill is determined by/proc/<pid>/oom_score. Each process has one value, which is calculated by the oom_badness () function of the Linux kernel. Next, read the badness () function carefully.

In the comments section of the badness () function, the following describes how to handle the badness () function:

1) we lose the minimum amount of work done
2) we recover a large amount of memory
3) we don't kill anything innocent of eating tons of memory
4) we want to kill the minimum amount of processes (one)
5) we try to kill the process the user expects us to kill, this algorithm has been meticulously tuned to meet the principle of least surprise... (be careful when you change it)

In general, Kill the minimum number of processes to obtain the maximum amount of memory, which is consistent with the process where Kill occupies the maximum memory.

/*
* The memory size of the process is the basis for the badness.
*/

Points = p-> mm-> total_vm;

The starting point of the score is the RAM memory actually used by the process. Note that SWAP is not included here, that is, OOM Killer is only related to the actual physical memory of the process, but not Swap, in addition, we can see that the more physical memory the process actually uses, the higher the score, and the higher the score, the easier it will be sacrificed.

/*
* Processes which fork a lot of child processes are likely
* A good choice. We add the vmsize of the childs if they
* Have an own mm. This prevents forking servers to flood
* Machine with an endless amount of childs
*/
...
If (chld-> mm! = P-> mm & chld-> mm)
Points + = chld-> mm-> total_vm;

This section indicates that the memory occupied by the child process is calculated to the parent process.

S = int_sqrt (cpu_time );
If (s)
Points/= s;
S = int_sqrt (run_time ));
If (s)
Points/= s;

This indicates that the CPU usage of a process is longer or the running time of the process is longer, the lower the score is, and the less Kill the process.

/*
* Niced processes are most likely less important, so double
* Their badness points.
*/
If (task_nice (p)> 0)
Points * = 2;

If the process has a low priority (nice value, positive value has a low priority, and negative value has a high priority), the Point doubles.

/*
* Superuser processes are usually more important, so we make it
* Less likely that we kill those.
*/
If (cap_t (p-> cap_effective) & CAP_TO_MASK (CAP_SYS_ADMIN) |
P-> uid = 0 | p-> euid = 0)
Points/= 4;

Super User processes have a low priority.

/*
* We don't want to kill a process with direct hardware access.
* Not only cocould that mess up the hardware, but usually users
* Tend to only have this flag set on applications they think
* Of as important.
*/
If (cap_t (p-> cap_effective) & CAP_TO_MASK (CAP_SYS_RAWIO ))
Points/= 4;

The process with direct access to the original device has a higher priority.

/*
* Adjust the score by oomkilladj.
*/
If (p-> oomkilladj ){
If (p-> oomkilladj> 0)
Points <= p-> oomkilladj;
Else
Points >>=- (p-> oomkilladj );

}

Each process has a oomkilladj parameter that can set the priority of the process to be killed. This parameter seems to have a greater impact on the Point. oomkilladj has a maximum of + 15 and a minimum value of-17, the larger the value, the more likely it is to be killed. Because this value is a shift operation, the impact is still relatively large.

Below I will write a small program experiment:

 #define MEGABYTE 1024*1024*1024 #include <stdio.h> #include <string.h> #include <stdlib.h> int main(int argc, char *argv[]){void *myblock = NULL;myblock = (void *) malloc(MEGABYTE);printf("Currently allocating 1GB\n");sleep(1);int count = 0;while( count < 10){ memset(myblock,1,100*1024*1024); myblock = myblock + 100*1024*1024; count++; printf("Currently allocating %d00 MB\n",count); sleep(10);  }  exit(0); }

The above program first applies for a 1g memory space, and then m as the unit, fill these memory space. Run three processes on a machine with 2 GB memory and 400 M Swap space. Let's take a look at the running results:

Test1, test2, and test3 applied for 1 GB of virtual memory space (VIRT) respectively. Then, every 10 s, the actual occupied RAM space increased by 100 MB (RES ).

When the physical memory space is insufficient, the OS begins to perform Swap and the available Swap space begins to decrease.

When there is no allocable memory, the test1 process is killed by the operating system. Dmesg we can see that the test1 process is killed by OS and oom_score is 1000.

The default value of oom_adj for all three processes is 0. Next we will experiment with the effect of oom_adj. Restart the three processes. Then we can see that the PID of test2 is 12640.

Run the following statement:

Echo 15>/proc/12640/oom_adj

After a while, we can see a sharp decrease in Swap space, basically OS OOM_Killer is about to start.

Unexpectedly, the 12640 process was killed.

Therefore, to prevent the desired process from being killed, you can set oom_adj of the process. Of course, some people will say that this is caused by oversales. Since Linux provides overcommit_memory to disable the overcommit feature, why not disable it. This has both advantages and disadvantages. Once overcommit is disabled, it means that MySQL cannot apply for a space that exceeds the actual memory. in MySQL, there are many dynamic applications for memory space. If the application fails, mySQL will Crash, which greatly increases the risk of MySQL downtime, which is also why Linux overcommit.

With the above analysis, we can easily see that if oom_adj is not set, MySQL will generally become the preferred object for OOM_Killer, because MySQL is generally the largest memory owner. As MySQL, how can we avoid the risk of being killed as much as possible? in the next chapter, we will analyze how to avoid OOM from the perspective of MySQL.