An issue caused by a process memory layout exception

an issue caused by a process memory layout exception

The previous period of time the business reflects a certain type of server update bash, SSH connected to the occasional login failed, the client spit error message as follows:

Figure-0

This version of Bash is customized for the Department, but the implementation has not changed the original logic, but added some monitoring functions, then where do these errors come from?

Is it bash's pot?

From the above error message can be guessed that the exception is bash in the startup process caused by the allocation of memory failure, it appears that the process is in some cases mistakenly a large amount of memory allocation, resulting in insufficient memory, to confirm that this thing is relatively simple, dynamic memory allocation to the system call this layer mainly two ways: BRK ( ) and mmap (), so it is possible to estimate whether there is a large memory allocation as long as the call of the two is counted.

Bash is started by sshd, so Strace followed the sshd process, and found that when the exception occurred, bash allocated very little memory, less than sometimes even only dozens of bytes will fail, almost can be concluded that bash is not an exception in memory usage, but during this period found a strange phenomenon , Bash has been using only BRK in allocating small memory, BRK () after the failure of the direct exit, the general program used by the libc malloc (or other similar malloc) will be combined with BRK and mmap to use "0", not brk a failure to allocate memory, check Looking at the source of bash, it is true that it does its own memory management based on BRK and does not use malloc or mmap.

But that's not the point, and the point is that even if you only use BRK, you won't be able to allocate dozens of bytes of memory.

Memory layout of the process

The memory layout of the process is structurally regular, specifically for the process on the Linux system, its memory space can generally be roughly divided into the following large sections of "1", from high memory to low memory arrangement:
1, the kernel State memory space, its size is generally fixed (can be adjusted at compile time), but the value of 32-bit system and 64-bit system is different.
2, the user state stack, the size is not fixed, can be adjusted with Ulimit-s, the default is generally 8M, from high address to low address growth.
3, mmap area, process vast memory space in the main part, can be from high address to low address extension (so-called flexible layout), but also from low to high extension (so-called legacy layout), see the process of the specific situation "2" "3".
4, BRK area, close to the data segment (even affixed), from low to high, but its size depends mainly on how the mmap growth, generally speaking, even if the 32-bit process extends in the traditional way, there are almost 1 GB of space (accurately TASK_SIZE/3-code segment Data segment, see AR Definition of Ch/x86/include/asm/processor.h) "4"
5, the data section, mainly in the process initialization and uninitialized global data sum, of course, compiler generated some auxiliary data structure, and so on, the size depends on the specific process, its position is close to the code snippet.
6, the code snippet, mainly is the process instruction, including the user code and the compiler generates the auxiliary code, its size depends on the specific program, but the starting position is fixed according to 32 bit or 64 bit general fixation (-fpic,-fpie, etc. except "5").

The above paragraphs (in addition to the code snippet data segment) Its starting position according to whether the system is randomize_va_space generally slightly change, between each paragraph so there may be random size interval, thousands of words than a picture:

Figure-1

So now the problem boils down to: why the BRK area of the target process is suddenly so small, check the memory layout of bash first:

Figure-2

The memory layout of this process has a large discrepancy from the general understanding, from the top down to the low memory to the high memory:
#1is the code snippet and data segment for the process, which is generally at the lowest point in the process memory space, but is now clearly mapped to a dynamic library at a lower level.
#2BRK area, the area is still close to the data segment, but the BRK and code snippets are also inserted into the dynamic library, and, more importantly, the BRK area to the direction of the high, dynamic library mapping of the area is very close, resulting in the BRK region in fact there is only a small space (0x886000- 0x7ac000).

This is not the memory layout we want, we want to grow as follows:

Figure-3

It's different. No, two bash processes are 64-bit, different from the former is sshd process the latter is I manually on the terminal up, manual cat/proc/self/maps see the next 64-bit cat process memory layout is also normal:

Figure-4

What about the sshd process?

Figure-5

SSHD process is not normal, and unexpectedly found that sshd is 32 bits, and then wrote a test program:

Figure-6

The program compiles to 32 bits on the target machine to reproduce the problem, and if it compiles to 64 bits everything is fine, and another discovery is that as long as the 32-bit process, their memory layout is "unhealthy".

Operating system of the pot?

To figure this out, we have to figure out the process that processes start in the kernel, and for the process of the user state, any process is forked out from the parent process and then executed Execve, and EXECV calls the corresponding loader (mainly elf loader) to code snippets, data segments, and dynamic connectors ( Ld.so, if required) to load into the corresponding location of the memory space, the completion of a direct jump to the portal of the dynamic connector (this first ignores statically linked programs), the other dynamic library is loaded by the dynamic library connector, it should be noted that whether the kernel load ld.so or ld.so load other dynamic libraries, All require mmap assistance, which is used to locate the location in the memory space.

Now let's see what's wrong with the kernel, the target system version is as follows, after consulting the Tlinux, the system is based on the CentOS 6.5:http://vault.centos.org/6.5/centosplus/source/spackages/ kernel-2.6.32-431.el6.centos.plus.src.rpm

Figure-7

First look at the Arch/x86/mm/mmap.c:arch_pick_mmap_layout () function, which is to initialize the MMAP related portals based on the settings of the process and the current system:

Figure-8

The Exec-shield is a security function switch, a series of enhancements to the buffer overflow attack led by Red Hat many years ago, and can be found in several connections 1, 2,3,4,exec Shield has been problematic for implementation and use, Also destroyed some of the old program compatibility "6", so has not been in the trunk, only used in the Redhat family 6.x and its derived system.

This feature has a switch/proc/sys/kernel/exec-shield, according to the instructions on the link "6", Exec-shield can be set to 0, 1, 2, 3, respectively: Force off/default off unless the executable program specifies that open/ Open by default unless the executable program specifies to close/force open.

Mm->get_unmapped_area is the final function called when a process needs to be mmap, and Arch_get_unmap_area () is used to search for the right position in the traditional way from low, Arch_get_unmapped_area_ Topdown () Flexible layout from the high start to search for the right position, the key point is 125 ~ 129 rows, Exec-shield introduced another special for 32-bit process memory allocation method, this way specifies if the memory to be allocated requires executable permissions , you should start searching for the right location from Mm->shlib_base, where the value of Shlib_base is shlib_base plus a small random offset, and the Shlib_base value is "7":

Figure-9

Notice that the address is before the code snippet for the 32-bit process (0x8048000), so this explains why the 32-bit process, its dynamic library is loaded into the low or even into the gap between BRK and data segments, this particular way of searching memory space is only for the memory that needs to execute the permissions, However, since the Elf loader is fragmented (pt_load) when loading the dynamic library, the first segment is assigned by the Mm->get_unmap_area () search for the appropriate location, and subsequent segments are forced to be placed after the first segment using Map_fixed. So the data segment is also mapped to the low. "9"

1641 line shows how to get the Get_area function from the MM structure when mmap, you can see that as long as Mm->get_unmmapped_exec_area is not empty, and the memory to be allocated requires executable permissions, it takes precedence mm->get_ Unmmapped_exec_area to search.

Figure-10

The above-described exec memory allocation is actually very easy to cause conflict, Redhat here also played a lot of patches, see.

The problem is not resolved

The above explanation explains why the memory layout of the 32-bit process is abnormal, but the problem here is that the 64-bit process is also affected when 32-bit processes are used to start 64-bit processes. To figure this out, you have to look at the fs/binfmt_elf.c:load_elf_binary () function, which is used to load the elf-formatted executable file in the current process and jump over it, which is shared by the 32-bit elf and the 64 elf (with the help of a more covert macros), the things that it does are summed up to include the following:
1, read and parse the Elf file contains a variety of information, key information such as code snippets, data segments, dynamic linker and so on.
2. Flush_old_exec (): Stops all threads in the current process, empties the current memory space, resets various states, and so on.
3, set the status of the new process, such as allocating memory space, initialization and so on.
4. Load the dynamic connector and skip past execution.

Figure-11

Now back to our problem, the current process is 32-bit, the 32-bit process on a 64-bit system requires kernel support, and when the kernel discovers that the elf is a 32-bit program, a flag is placed inside the task, which in the Load_elf_binary () function calls S in 740 rows Et_personality () is cleared, so at 721, the current process still considers itself 32-bit, and flush_old_exec () did something about it, see: Fs/exec.c:flush_old_exec ()

Figure-12

Note that 1039 of these lines, BPRM->MM represents the new memory space (the old is still in, but immediately release and switch the new), where the new memory space needs to be set, see: Fs/exec.c:exec_mmap ()

Figure-13

We can see that in the current process or 32 bits, the kernel initializes the new memory space, causing arch_pick_mmap_layout () to incorrectly assign the Arch_get_unmaped_exec_area to bprm->mm- >get_unmapped_exec_area This member variable, although the Load_elf_binary () function in Figure 11 is in line 748, the 32-bit flag is emptied and then called Again set_up_new_exec (), Arch_get _unmapped_exec_area (), but Arch_get_unmaped_exec_area () does not empty mm->get_unmapped_exec_area this variable, resulting in EXECV although the process is 64-bit, But still with Mm->shlib_base here as the starting address to search memory space for the dynamic library to use, oops.

Solution Solutions

The most direct and reliable approach is to enter the Arch_pick_mmap_layout (), the first place Mm->get_unmapped_exec_area null, but this will modify the kernel, the user state to circumvent the words have the following ways:
1, set Ulimit-s Unlimited, and set the Exec-shield to 0 or 1, then the process, this, because the user-state stack is infinitely long, the kernel can only traditional way to the 32-bit process to allocate memory, will not fall into the Exec-shield pit.
2, the randomize_va_space ban, but this practice just buried the head into the sand.

In general, the above two types of user-based avoidance scheme is basically where the pain to paste, not to solve the problem (and there is a security risk), step back, do not use 32-bit process to start 64-bit process is also relatively safe point.

Reference

"0" Https://en.wikipedia.org/wiki/C_dynamic_memory_allocation
"1" Https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Tuning_and_Optimizing_Red_Hat _enterprise_linux_for_oracle_9i_and_10g_databases/sect-oracle_9i_and_10g_tuning_guide-growing_the_oracle_sga_ To_2.7_gb_in_x86_red_hat_enterprise_linux_2.1_without_vlm-linux_memory_layout.html
"2" Understanding the Linux kernel, page 819, flexible memory region layout:https://books.google.com.hk/books?id= h0lltxyj8aic&pg=pt925&lpg=pt925&dq=linux+flexible+memory&source=bl&ots=go7riyb8hr&sig= Pirb5pswdhfhsljy57eksxs3abw&hl=en&sa=x&ved=0ahukewjpkfa-2_rrahvgfjqkhcetdsuq6aeitdah#v=onepage &q=linux%20flexible%20memory&f=false
"3" https://gist.github.com/CMCDragonkai/10ab53654b2aa6ce55c11cfc5b2432a4
"4" http://lxr.free-electrons.com/source/arch/x86/include/asm/processor.h#L770
"5" https://access.redhat.com/blogs/766093/posts/1975793
"6" https://lwn.net/Articles/31032/
"7" https://lwn.net/Articles/454949/
"8" http://lxr.free-electrons.com/source/fs/binfmt_elf.c#L549
"9" http://lxr.free-electrons.com/source/fs/binfmt_elf.c#L563
"10" Similar problem: https://bugzilla.redhat.com/show_bug.cgi?id=870914 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=522849

An issue caused by a process memory layout exception