Linux thread (process) number limit analysis

Tag: State user Sys occupies system problem value development check

1. Source of the problem


On the company's online environment, MQ could not accept the exception of the message, and the operation and the developer temporarily switched on to another server's MQ after recovery. At the same time, operation and maintenance staff feedback on the server that has the problem many basic commands can not run, the following error occurred:



2. Preliminary cause analysis and resolution


Let ops brother on the service to view the memory, CPU, network, IO and other basic information are normal. So he went to the operation of the server to see a bit, the following is the Slabtop–s C operation results, the initial cause of the problem seems to appear:



If you see this you can not see anything unusual, the following content you may not be interested in, haha ...


Task_struct is the kernel to the process of management units, through the Slub (slab upgrade, if you do not understand slub or affect the following content, as long as the slab on the line) for node management, the normal load of the service should not appear task_ The slub structure of a struct occupies the largest amount of memory, which means that a large number of processes are open on this server (the Linux kernel is a single unit for processes and threads, so do not tangle with this, which may be followed by process and thread mixing).


Through this information, the brothers found that the server has nearly 30,000 threads, but also to locate the problem of the network element (a new classmate of the code is not review directly online, there is a bug that triggered the exception to create a large number of threads).


The problem seems to end here, but as a feeling of the programmer, this is just a beginning (brother's feelings during the day are the tedious work of grinding, only in this late night exclusive ... ）


3. Limitations on the number of Linux threads


3.1 Application Layer test code


#define MEMSIZE (1024 * 1024 * 256)

void thread (void)

{

Sleep (100);

Return

}



int main ()

{

pthread_t ID;

int ret;

int num = 0;

while (1) {

ret = pthread_create (&id, NULL, (void*) thread, NULL);

++num;

if (ret! = 0)

Break

}

printf ("Pthread_create fail with ret=%d, Total num=%d\n", ret, num);

Sleep (100);

return 0;

}


With strace tracking, the problem is found in the Copy_process function, and the rest of the work is to analyze the cause of the copy_process return exception.


3.2 Reverse Analysis


This time the inverse analysis is the simplest direct, can directly locate the cause of the problem.


First, by strace analysis, the system call to find the problem is the clone function.


Sys_clone->do_fork->copy_process. Kernel function Analysis Tool this time the trial of Systemtap, below is no any aesthetic STAP code, will look at it


Probe kernel.statement ("* @kernel/fork.c:1184")

{

printf ("in kernel/fork.c 1184\n");

}

Probe kernel.statement ("* @kernel/fork.c:1197")

{

printf ("in kernel/fork.c 1197\n");

}

Probe kernel.statement ("* @kernel/fork.c:1206")

{

printf ("1113.www.qixoo.qixoo.com in Kernel/fork.c 1206\n");

}

Probe kernel.statement ("* @kernel/fork.c:1338")

{

printf ("in kernel/fork.c 1338\n");

}

Probe kernel.statement ("* @kernel/fork.c:1342")

{

printf ("in kernel/fork.c 1342\n");

}

Probe kernel.statement ("* @kernel/fork.c:1363")

{

printf ("in kernel/fork.c 1363\n");

}

Probe kernel.statement ("* @kernel/fork.c:1369")

{

printf ("in kernel/fork.c 1369\n");

}

Probe kernel.statement ("* @kernel/fork.c:1373")

{

printf ("in kernel/fork.c 1373\n");

}

Probe kernel.function ("Copy_process"). return

{

printf ("Copy_process return%d\n", $return)

}

function Check_null_pid:long (Addr:long)

{

struct PID *p;

p = (struct pid*) this->l_addr;

if (p = = NULL)

This->__retvalue = 0;

Else

This->__retvalue = 1;

}

Probe kernel.function ("Alloc_pid")

{

printf ("Alloc_pid init\n");

}

Probe kernel.statement ("* @kernel/pid.c:301")

{

printf ("Alloc_pid 301\n");

}

Probe kernel.statement ("* @kernel/pid.c:312")

{

printf ("Alloc_pid 312\n");

}

Probe kernel.function ("Alloc_pid"). return

{

printf ("Alloc_pid return%ld\n", Check_null_pid ($return));

}


The problem is found in alloc_pid failure, the analysis of kernel code, which is limited by the Kernel.pid_max parameter.



After the parameter is set to 100000, run again.



Continue to trace through strace, this discovery problem is in the Mprotect function



This problem is due to the mmap number limit for a thread, subject to the Vm.max_map_count parameter.



After the parameter is raised to 100000, the number of threads is significantly increased after running again.


In fact, there is a parameter Kernel.threads-max limit, because the system defaults to set this parameter to 800000, very large, so the effect of this parameter has not been preserved.


And then make a base of the relevant parameters are set to 800000, the result of memory exhaustion, the system is not responding directly ....


3.3 Forward Analysis


Direct analysis of Copy_process code


Copy_process


3.3.1 Memory Limit


Dup_task_struct–>alloc_task_struct_node/alloc_thread_info_node/arch_dup_task_struct–>kmme_cache_alloc_ Node (SLUB.C) –>slab_alloc_node–>


"Config_memcg_kmem"//Here is also a pit, Docker this based on Cgroup will also affect, may be allocated to slub the memory is not enough with the thread limit


Specific functions:


Alloc_pages-->__memcg_kmem_newpage_charge–>memcg_charge_kmem–>__res_counter_charge–>res_counter_ Charge_locked


3.3.2 Threads-max parameter Limits


if (nr_threads >= max_threads)//Threads-max parameter influence


3.3.3 Pid_max parameter Limits


Alloc_pid–>alloc_pidmap//pid_max parameter influence


3.3.4 single-Process memory limit


The number of threads for a single process, limited by the vm.max_map_count limit


4. Summary


/proc/sys/kernel/pid_max #操作系统线程数限制


/proc/sys/kernel/thread-max #操作系统线程数


Max_user_process (ulimit-u) #系统限制某用户下最多可以运行多少进程或线程


/proc/sys/vm/max_map_count #单进程mmap的限制会影响当个进程可创建的线程数


/sys/fs/cgroup/memory/${cgroup}/memory.kmem #单个docker kernel memory limits, which can affect the application of slab nodes such as task_struct, indirectly affecting the number of threads that can be created

Linux thread (process) number limit analysis