In-depth analysis of Linux heap memory management (II) and in-depth analysis of Memory Management

In-depth analysis of Linux heap memory management (II) and in-depth analysis of Memory Management

In-depth analysis of Linux heap memory management

(Lower half)

Author @ location, Alibaba Cloud universal security

0 preface Review

In the previous article (see the bottom of the article), I introduced in detail the basic concepts and relationships involved in heap memory management, it also focuses on the implicit linked list Technology Used in heap chunk allocation and release policies. Based on the previous introduction, we know that using an implicit linked list to manage memory chunk will always involve memory traversal, with extremely low efficiency. Glibc malloc introduces the display linked list technology to improve the efficiency of heap memory allocation and release.

The so-called display linked list is a commonly used linked list in data structures. In essence, the linked list concatenates "nodes" with the same attributes to facilitate management. In glibc malloc, these linked lists are collectively referred to as bin. The "Node" in the linked list is each chunk. The common attribute of the node is: 1) free chunk; 2) the chunks in the same linked list are of the same size (for details, see the following section ).

 

1 bin Introduction

As mentioned above, bin is a linked table data structure that records free chunk. For free chunks of different sizes, the system divides the bin into four categories: 1) Fast bin; 2) Unsorted bin; 3) Small bin; 4) Large bin.

 

There are two data structures used to record bin in glibc:

FastbinsY: This is an array used to record all fast bins;

Bins: this is also an array used to record all bins except fast bins. In fact, there are a total of 126 bins, which are:

Bin 1 is unsorted bin;

Bin 2 to 63 is small bin;

Bin 64 to 126 is large bin.

The specific data structure is defined as follows:

Struct malloc_state { ...... /* Fastbins */ Mfastbinptr fastbinsY [NFASTBINS]; ...... /* Normal bins packed as described abve */ Mchunkptr bins [NBINS * 2-2]; // # define NBINS 128 ...... }; Mfastbinptr definition: typedef struct malloc_chunk * mfastbinptr; Mchunkptr definition: typedef struct malloc_chunk * mchunkptr;

 

Drawing is more intuitive:

 

Figure 1-1 bins Classification

 

So how is the free chunks in bins linked together? Review the data structure of malloc_chunk:

Struct malloc_chunk { /* # Define INTERNAL_SIZE_T size_t */ INTERNAL_SIZE_T prev_size;/* Size of previous chunk (if free ).*/ INTERNAL_SIZE_T size;/* Size in bytes, including overhead .*/ Struct malloc_chunk * fd;/* The two pointers only exist in free chunk */ Struct malloc_chunk * bk;   /* Only used for large blocks: pointer to next larger size .*/ Struct malloc_chunk * fd_nextsize;/* double links -- used only if free .*/ Struct malloc_chunk * bk_nextsize; };

The fd and bk pointers point to the forward or backward chunk in the linked list to which the current chunk belongs.

 

2 Fast bin

 

Since there is fast bin, there must be a fast chunk -- chunk with a size of 16 to 80 bytes is called fast chunk. To facilitate the subsequent description, the following conventions are made on the chunk size:

1) when it comes to chunk size, it indicates the actual overall size of the malloc_chunk;

2) When it comes to chunk unused size, it indicates the actual available size of the secondary members such as prev_size, size, fd and bk in the malloc_chunk. Therefore, for free chunk, the actual available size is always 16 bytes less than the actual total size.

 

In the process of memory allocation and release, fast bin is the fastest operation speed in all bin. The following describes some features of fast bin:

1) Number of fast bins-10

2) Each fast bin is a single-chain table (only using the fd pointer ). Why is a single-chain table used? In fast bin, both adding and removing fast chunks operate on the end of the chain table instead of a Middle fast chunk. More specifically, the LIFO (post-in-first-out) algorithm: The add operation (free memory) adds the new fast chunk to the end of the linked list, and the delete operation (malloc memory) is to delete the fast chunk at the end of the linked list. Note that in order to implement the LIFO algorithm, each fastbin element in the fastbinsY array points to the rear end (end Node) of the linked list ), the end node uses its fd pointer to indicate the forward node, and so on, as shown in 2-1.

3) chunk size: the fast chunk size contained in the 10 fast bins is arranged in 8 byte increments, that is, all the fast chunk size in the First fast bin is 16 bytes, the second fast bin is 24 bytes, and so on. During malloc initialization, the maximum fast chunk size is set to 80 bytes (the chunk unused size is 64 bytes ), therefore, chunks of 16 to 80 bytes are classified into fast chunks by default. Details 2-1 are shown.

4) The free chunk is not merged. Since the original intention of designing fast bin is to quickly allocate and release small memory, the system always sets the P (unused flag bit) of chunk of fast bin to 1, in this way, even if a chunk in the fast bin is adjacent to the same free chunk, the system does not perform the automatic merge operation, but retains the two. Although this may cause extra fragmentation problems, it is not a secret.

5) malloc (fast chunk) operation: that is, the size of your request through malloc falls within the size range of fast chunk (note: the size of your request plus 16 bytes is the actual chunk size of the memory ). During initialization, the maximum memory size supported by fast bin and all fast bin linked lists are empty. Therefore, when malloc is used to apply for memory, even if the requested memory size belongs to the memory size of the fast chunk (16 to 80 bytes), it will not be processed by the fast bin, but will be passed down to the small bin for processing, if small bin is empty, it will be handled by unsorted bin:

/* Maximum size of memory handled in fastbins .*/ Static INTERNAL_SIZE_T global_max_fast;   /* Offset 2 to use otherwise unindexable first 2 bins */ /* Here SIZE_SZ is sizeof (size_t), where the 32-bit system is 4 and 64-bit is 8, fastbin_index is used to quickly calculate which fast bin the size should belong to based on the size of malloc, that is, the index of the fast bin. Because chunk in fast bin starts from 16 bytes, all operations in 8 bytes (32-bit system as an example) are reduced by 2*8 = 16! */ # Define fastbin_index (sz )\ (Unsigned int) (sz)> (SIZE_SZ = 8? 4: 3)-2)     /* The maximum fastbin request size we support */ # Define MAX_FAST_SIZE (80 * SIZE_SZ/4)   # Define NFASTBINS (fastbin_index (request2size (MAX_FAST_SIZE) + 1)

 

So where is fast bin? How to initialize it? When we call malloc (fast bin) for the first time, the system executes the _ int_malloc function. This function first finds that the current fast bin is empty and forwards it to small bin for processing, in addition, if small bin is empty, the malloc_1_lidate function is called to initialize the malloc_state struct. The malloc_1_lidate function mainly implements the following functions:

A. First, judge whether the fast bin in the current malloc_state struct is null. If it is null, the entire malloc_state is not fully initialized and the malloc_state needs to be initialized.

B. the initialization operation of malloc_state is completed by the malloc_init_state (av) function. This function first initializes all bins except fast bin (create a double-stranded table. For details, see the introduction of small bins ), then initialize fast bins.

Then, when you execute the malloc (fast chunk) function again, if the data related to fast bin is not empty, use fast bin (see ※1 in the following code ):

Static void * _ Int_malloc (mstate av, size_t bytes) { ...... /* If the size qualifies as a fastbin, first check corresponding bin. This code is safe to execute even if av is not yet initialized, so we Can try it without checking, which saves some time on this fast path. */ // When you execute malloc (fast chunk) for the first time, the value is false because get_max_fast () is 0. If (unsigned long) (nb) <= (unsigned long) (get_max_fast ())) { ※1 idx = fastbin_index (nb ); Mfastbinptr * fb = & fastbin (av, idx ); Mchunkptr pp = * fb; Do { Victim = pp; If (victim = NULL) Break; } ※2 while (pp = catomic_compare_and_exchange_val_acq (fb, victim-> fd, victim ))! = Victim ); If (victim! = 0) { If (_ builtin_ct (fastbin_index (chunksize (victim ))! = Idx, 0 )) { Errstr = "malloc (): memory upload uption (fast )"; Errout: Malloc_printerr (check_action, errstr, chunk2mem (victim )); Return NULL; } Check_remalloced_chunk (av, victim, nb ); Void * p = chunk2mem (victim ); Alloc_perturb (p, bytes ); Return p; } }

After obtaining the first chunk from the fast bin, the system removes the chunk from the corresponding fast bin and returns the address to the user. See the preceding Code ※2.

6) free (fast chunk) operation: this operation is very simple and mainly divided into two steps: the chunksize function is used to obtain the chunk size of the pointer based on the input address pointer; then, obtain the fast bin to which the chunk belongs Based on the chunk size, and add the chunk to the end of the chain of the fast bin. The entire operation is completed in the _ int_free function.

Shows the overall operation of Fast bins (array fastbinsY) in main arena:

Figure 2-1 fast bin

 

3 Unsorted bin

When small or large chunks are released, if the system does not add them to the corresponding bins (why, under what circumstances will this happen? The system adds these chunks to the unsorted bin. Why? This is mainly to enable the "glibc malloc mechanism" to have a second chance to reuse the recently released chunk (the first chance is the fast bin mechanism ). The unsorted bin can be used to accelerate the memory allocation and release operations, because the entire operation no longer requires additional time to find the appropriate bin.

Unsorted bin has the following features:

1) Number of unsorted bin: 1. Unsorted bin is a cyclic double-stranded table composed of free chunks.

2) Chunk size: In unsorted bin, there is no limit on the chunk size. Any chunk of any size can belong to unsorted bin. This is the special case in the preface. However, this is not the only case. We will introduce it later.

 

4 Small bin

A chunk smaller than 512 bytes is called small chunk, and small bin is used to manage small chunk. In terms of memory allocation and release speed, small bin is faster than larger bin, but slower than fast bin.

The features of Small bin are as follows:

1) number of small bin: 62. Each small bin is also a cyclic double-stranded table composed of the corresponding free chunk. At the same time, Small bin uses the FIFO algorithm: the memory release operation adds the newly released chunk to the front end (front end) of the linked list ), the allocation operation obtains the chunk from the rear end (tail end) of the linked list.

2) chunk size: What is the size of all chunks in the same small bin? And the chunk size in the first small bin is 16 bytes. The chunk size in each small bin is increased by 8 bytes in sequence, that is, the chunk of the last small bin is 16 + 62*8 = 512 bytes.

3) Merge operation: the adjacent free chunks need to be merged to form a large free chunk. For more information, see free (small chunk.

4) malloc (small chunk) operation: similar to fast bins, all the initial small bin is empty, so before initialization of these small bin, even if the memory size of the user request belongs to the small chunk, It is not handled by the small bin, but by the unsorted bin. If the unsorted bin cannot be processed, glibc malloc traverses all the subsequent bins in sequence to find the first bin that meets the requirements. If none of the bin meets the requirements, the top chunk will be used instead. If the top chunk is not large enough, so we can expand the top chunk to meet the requirements. (Do you still remember the problems left in the Top Chunk in the previous article? The answer is here ). Note that subsequent bins traversal and subsequent operations are also used by large bin. Therefore, we will introduce this part in the malloc operation of large bin.

So how does glibc malloc initialize these bins? Because these bins belong to the malloc_state struct, these bins are initialized when initializing malloc_state. The Code is as follows:

Malloc_init_state (mstate av) { Int I; Mbinptr bin;   /* Establish circular links for normal bins */ For (I = 1; I <NBINS; ++ I) { Bin = bin_at (av, I ); Bin-> fd = bin-> bk = bin; } ...... }

Note that in the malloc source code, set the index value of the first Member in the bins array to 1 instead of the commonly used 0 (in the bin_at macro, I is automatically reduced by 1 for processing ...). From the code above, we can see that glibc malloc points all bin pointers to itself during initialization -- this indicates that these bins are empty.

After that, when calling malloc (small chunk) again, if the small bin corresponding to the chunk size is not empty, the small chunk will be obtained from the small bin linked list, otherwise, it will be handed over to the unsorted bin and the subsequent logic for processing.

5) free (small chunk): When the small chunk is released, check whether the adjacent chunk of the chunk is free. If yes, perform the merge operation: merge these chunks into new chunks, remove them from the small bin, and add the new chunks to the unsorted bin.

 

5 Large bin

A chunk larger than 512 bytes is called a large chunk. large bin is used to manage these large chunks.

The features of Large bin are as follows:

1) number of large bin: 63. Large bin is similar to small bin, but you only need to pay attention to two points: first, the size of each chunk in the same large bin can be different, but it must be in a given range (exception 2 ); second, large chunk can be added or deleted to any location in large bin.

Among the 63 large bins, the first 32 large bin are separated by a 64-byte step, that is, the chunk size in the first large bin is 512 ~ 575 bytes. The chunk size in the second large bin is 576 ~ 639 bytes. The next 16 large bin values are separated by the step size of 512 bytes. The subsequent 8 bin values are separated by the step size of 4096, and the subsequent 4 bin values are separated by 32768 bytes; the next two bins are separated by 262144 bytes, and the remaining chunks are placed in the last large bin.

Since the size of each chunk in the same large bin is not necessarily the same, in order to speed up memory allocation and release, all chunks in the same large bin are arranged in the descending order according to the chunk size: the largest chunk is placed at the front end of the linked list, and the smallest chunk is placed at the rear end.

2) Merge operation: similar to small bin.

3) malloc (large chunk) operation:

The operations before Initialization is complete are similar to small bin. Here we mainly discuss the operations after large bins Initialization is complete. First, determine which large bin the user request belongs, then, determine whether the size of the largest chunk in the large bin is greater than the size of the user request (you only need to compare the size of the front end in the linked list ). If the value is greater than, the system traverses the large bin from the rear end, finds the first chunk with the same size or proximity, and assigns it to the user. If the chunk is larger than the size requested by the user, the chunk is split into two chunks: the former is returned to the user, and the size is equivalent to the size requested by the user; add the remaining part as a new chunk to the unsorted bin.

If the size of the largest chunk in the large bin is smaller than the size requested by the user, check whether there is a chunk meeting the requirements in the subsequent large bin, however, it should be noted that, in view of the large number of bins (the chunk in different bin is very likely to be in different memory pages ), if you traverse the chunk in each bin according to the method described in the previous section, the memory page may be interrupted multiple times, seriously affecting the retrieval speed, therefore, glibc malloc designed the Binmap struct to help increase the bin-by-bin search speed. Binmap records whether each bin is empty. bitmap can be used to avoid searching empty bin. If the next non-Empty large bin is found through binmap, the chunk is allocated according to the method in the previous section. Otherwise, the top chunk is used to allocate the appropriate memory.

4) Free (large chunk): similar to small chunk.

After understanding the above knowledge and combining 5-1, it is not difficult to understand the processing logic of various bins:

6. Conclusion

Now, all the display linked list technologies involved in glibc malloc have been introduced. Given the limited space and energy, this article does not cover all the technical details in detail, but I believe that with these knowledge points to study glibc malloc again, it will be able to get twice the result with half the effort.

In addition, as far as I personally know about Heap Overflow attacks, I have mastered the above knowledge and understand most Heap Overflow Attack technologies. Therefore, the subsequent articles will introduce the implementation principles of each attack technology in detail based on these knowledge.

Old rule: if you have any mistakes, please make sure you are correct!

 

Review of previous articles

[Linux heap memory management in-depth analysis (I )]

 

Author: Go @ aliyunju security. For more security technical articles, click the aliyunju security blog.