Compiling, linking, and loading under Linux

--"Programmer's self-cultivation" reading notes

Compilation process

The process of compiling the source code into executable files using GCC under Linux can be decomposed into 4 steps, namely preprocessing (prepressing), compiling (compilation), assembly (Assembly), and linking (linking). A simple Hello Word program compiles the process as follows:

1. Pretreatment

First the source code file (. c/.cpp) and the associated header file (. h/.hpp) are pre-compiled by the preprocessor CPP into. i files (c + + for. II). The preprocessing commands are:

Gcc–e Hello.c–o hello.i

The pre-compilation process primarily handles pre-compiled instructions that start with # in the source code, with the following main processing rules:

• Delete all # define, and expand all macro definitions;
• Handles all conditional compilation directives, such as # if, #ifdef等;
• Processes the # include precompiled directive, inserting the contained file into the location of the precompiled instruction. The process is recursive, and the included files may contain other files as well.
• Delete all comments//and/**/;
• Add line numbers and file identifiers, such as # # "HELLO.C" 2, so that compile-time compiler generates line number information for debugging and can display line number information when compiling errors or warnings are generated at compile time;
• Keep all #pragma compiler directives because the compiler needs to use them.

1. Compile

The compilation process is a series of lexical analysis, parsing, semantic analysis and optimization of the pre-processed files into corresponding assembly code files (. s). The compiled commands are:

Gcc–s Hello.i–o Hello.s

or output the assembly code file directly from the source file:

Gcc–s Hello.c–o Hello.s

The current version of GCC merges the precompiled and compiled two steps into one step, which is done by the program CC1 (c + + is Cc1plus).

1. Assembly

The assembly is the conversion of the assembly code into a command that can be executed by the machine, generating the target file (. o), which can be completed by a one by one translation of the assembly instruction and the machine instruction. The Assembly commands are:

Gcc–c Hello.s–o hello.o

Or output the target file directly from the source file:

Gcc–c Hello.c–o hello.o

1. Link

The link is that the linker LD assembles the various target files together, resolves the symbolic dependencies, the library dependencies, and generates the executable file. The linked commands are:

Ld–static crt1.o crti.o crtbegint.o hello.o–start-group–lgcc–lgcc_eh–lc-end-group crtend.o CRTN.O

In general, we use a single command to complete the above 4 steps:

GCC hello.c

In fact, GCC is just a package of other programs that invoke precompiled compiler cc1, assembler as, and linker ld according to different parameters.

Target file

The executable file format for Linux is elf (executalbe linkable format), including executables, relocatable files (target files. O, static Libraries. A), shared target files (Dynamic library. So), core dump files (kernel dump). The elf target file is structured as follows:

One of the important structures in Elf files related to paragraphs is the Segment Table (section Header table), which describes all the segments contained in the Elf file, such as the name of each segment, its length, the offset in the file, the read-write permission, and other attributes of the segment. We can use the Readelf tool to view the segments of the elf file:

Several of the more important paragraphs are as follows:

Segment Name	Description
. text	Store the compiled machine instructions
. Data	Holds initialized global static variables and local static variables
. rodata	Store read-only data, such as global const variables, string constants
. BSS	To store uninitialized global static variables and local static variables
. symtab	Symbol table, log symbol information
. rel.xxx	Reposition table, record. XXX segment need to reposition the anchor symbol

The essence of the linking process is to glue a number of different target files into a single whole, and the object file is actually a reference to the address between the target files, that is, the address of the function and the variable. In the link, we refer to the functions and variables collectively as symbols (symbol), function names and variable names are symbolic names (symbol name), we can consider the symbol as a link in the binder, the entire link process is based on the symbol can be done correctly. Each target file will have a symbol table, which is the. Symtab segment, which records all the symbols used in the target file. Each defined symbol has a corresponding value, called the symbol value, and for variables and functions, the symbolic value is their address. We can use the Readelf tool to view all the symbol information in the symbol table:

Static links

Linker chaining is the process of merging several input target files into one output file. The method of merging is simply to combine segments of the same nature, such as the. Text segment of the input file into a. Text segment, followed by the. Data segment,. BSS segment, and so on, as shown in:

A linker typically uses a method called a two-step link:

1. Space and address assignment. The linker scans all of the input target files, merges their segments, calculates the length and position of each segment in the output file, establishes a mapping relationship, and collects all the symbol definitions and symbol references from the symbol table of the input target file, unifying them into a global symbol table.
2. Symbolic parsing and relocation. Use all the information collected in the previous step, read the data in the middle of the input file, reposition the information, and do symbolic resolution and relocation, adjust the location of the code, etc., this step is the core of the link process.

Before linking, the virtual address of all segments in the destination file is 0 because the virtual space is not yet assigned. After the link, each segment of the output file is assigned to the corresponding virtual address. Similarly, when the linker merges the middle of the input file, it can calculate the new offset of the symbol in the symbol table, and the final virtual address of the symbol can be computed by the virtual address of the segment and the offset of the symbol in the segment.

Relocation is the process of connecting symbol references and symbol definitions. In the target file, there is a structure called the Relocation table (relocation table) that is designed to hold information related to relocation, and for each elf segment to be relocated has a corresponding relocation table, and a relocation table is often a segment in the Elf file. For example, if both the text and. Data fields have been relocated, there will be a corresponding relocation table. Rel.text segment and. Rel.data segment. Each place to be relocated is called a reposition entry (relocation Entry), and the offset of the relocated entry indicates the position of the entry in the segment to be relocated. During relocation, each relocation entry is a reference to a symbol, and when the linker needs to reposition a reference to a symbol, it determines the target address of the symbol, and the linker will then find the global symbol table that consists of all the symbol tables of the input target file, and find the corresponding symbol to reposition it. Let's look at the following symbol table:

The symbol shared and swap of type Global is und, and this undefined symbol is because there are relocation entries for them in the destination file. So after the linker has scanned all of the input target files, all these undefined symbols should be found in the global symbol table, otherwise the linker will report the symbol undefined error.

In static links, in addition to the source code generated by the target file, you also need to link to other static libraries, such as the C language static library libc. In fact, a static library can simply be regarded as a set of target files, that is, many of the target files are compressed after the formation of a file. We can use the AR tool to see which target files are included in the static library:

When the linker is linking the static library with the target file, only the symbol defined in a target file in the static library is referenced, and the target file is linked in.

Loading

The executable file can only be executed by the CPU after it is loaded into memory. The operating system creates a process and then loads the appropriate executable file and executes it, and the process starts with only 3 things to do:

1. Create a separate virtual address space. Creating a virtual space is essentially allocating a page directory, and the mapping of the virtual space to the physical memory is then set when the subsequent page error occurs.
2. Reads the executable header (program header Table) and establishes a mapping between the virtual space and the executable file. When the operating system captures a page fault, the mapping relationship knows where the currently required pages are located in the executable file. This mapping is mapped by Segment (Segment), where a segment in the process virtual space is called the Virtual memory area (vma,virtual).
3. Set the execution register of the CPU to the entry address of the executable file and start running. The elf file header holds the entry address, and the operating system transfers control to the process by setting the CPU instruction register, and the process begins execution.

A segment in the process virtual space is called the Virtual memory area (vma,virtual), a VMA maps the executable according to a segment, and in the elf file merges the same section with the same permissions into a segment, The system officially maps executables according to segment rather than sections. From the section perspective, the Elf file is a connection view (linking views), from the perspective of segment is the execution view (Executiong view). When we talk about Elf loading, paragraph refers specifically to segment, whereas in other cases, paragraph refers to section.

The segment information for elf files is stored in the executable header (program header Table), which describes how the elf file is mapped to the virtual space of the process by the operating system and can be viewed through the Readelf tool segment:

In, two segment of type load need to be mapped, and we can see which sections are merged into these two segment.

VMA in addition to being used to map the various segment in the executable, the heap, stack, and so on, which the process uses to execute, also exist in the form of VMA. We can view the distribution of process virtual space:

The operating system manages the virtual space of the process by dividing the process space into a VMA, and the basic principle is to map the same image file to a VMA with the same permission attributes. A process can basically be divided into the following VMA areas:

• Code VMA, permissions are read-only, executable, and have image files.
• Data VMA, permissions can read and write, executable, with image files.
• Heap VMA, permissions can read and write, executable, no image files, anonymous, can be expanded.
• Stack VMA, permissions can read and write, non-executable, no image files, anonymous, can be scaled down.

The virtual space for a common process is as follows:

Dynamic links

The basic idea of dynamic linking is to split the program into relatively separate parts according to the module, and link them together to form a complete program when the program is running, instead of connecting all the program modules to a single executable file like a static link. Elf dynamic Link files are called Dynamic shared objects (Dso,dynamic shared object), or shared objects, and they are generally files with an. so extension. Compared to static links, dynamic linkage has two advantages, one is that the shared object in disk and memory only one copy, save space; second, when you upgrade a shared module, you only need to replace the destination file without having to relink all the programs.

The final mount address of the shared object is not deterministic at compile time, but at load time, the loader dynamically allocates a large enough virtual address space to the corresponding shared object, based on the idle situation of the current address space. In order to be able to load the shared object at any address, the reference to all absolute addresses is not relocated at the time of the connection, and this step is deferred until the load is completed, that is, the load is relocated. At the same time, in order to realize the shared module's instruction part is shared among several processes, the shared instruction part cannot change because of the loading address, the solution is to separate the parts of the instruction that need to be modified, and put together the data parts so that the instruction part can remain unchanged. The data section can have a copy in each process, which is the technology of address-independent code (pic,position-independent), and we use the-fpic parameter in GCC to generate address-independent code. The symbol reference inside the module uses a relative address, so this instruction does not need to be relocated, whereas for symbolic references outside of the module, a global offset table (Got,global offset table) is established in the data segment, and the code is indirectly referenced by the corresponding item in the GOT. References to got also use relative addresses, the basic mechanism is as follows:

In the case of dynamic linking, when the operating system maps the executable, it launches a dynamic linker (Linker), Dynamic linker ld.so is actually a shared object, the operating system is also mapped to load it into the address space of the process, and give control to the portal address of the dynamic linker, the dynamic linker began to perform a series of its own initialization operations, and then according to the current environment parameters, to start the dynamic link to the executable file, when all the dynamic After the link is completed, the dynamic linker transfers control to the entry address of the executable file and the program starts to execute formally.

The most important structure in the dynamic Link elf is the. Dynamic segment, which holds the basic information required by the dynamic linker, such as which shared objects, the location of the dynamic link symbol, the location of the dynamic Link relocation table, the address of the shared object initialization code, and so on. Use the Readelf tool to view the contents of the. Dynamic segment:

You can also use the LDD tool to see which shared libraries a program or shared library relies on:

To represent the symbolic import and export relationships between dynamic link modules, the ELF has a special field called the dynamic symbol table that holds this information, This segment is often called. Dynsym, which only stores symbols associated with dynamic links, statically linked symbol tables. Syntab Save all the symbols, the general dynamic Link module has two symbol tables at the same time. You can use the Readelf tool to view the dynamic symbol table for elf files:

Dynamic linking is basically divided into 3 steps:

1. Dynamic linker bootstrap. When the operating system gives the process control to the dynamic linker, the bootstrap code of the dynamic linker begins execution. The bootstrap code obtains the relocation tables and symbol tables of the dynamic linker itself, repositioning them to use their own global variables and static variables.
2. Loads the shared object. After the bootstrap is complete, the dynamic linker merges the symbol tables of the executable file and the linker itself into a global symbol table. The linker then begins to look for shared objects on which the executable file depends, and puts the names of those shared objects into a Mount collection. The linker starts by taking the name of a desired shared object from the collection, opens the appropriate file and reads the elf file header and the. Dynamic segment, and maps its corresponding code snippet and data segment to process space. If the Elf shared object also relies on other shared objects, the shared objects that are dependent are placed in the Mount collection. So the loop knows that all dependent shared objects are loaded in.
3. Relocation and initialization. The linker begins to iterate through the executable file and the relocation table for each shared object, correcting each location in their got that needs to be relocated. After relocation is complete, if a shared object has a. Init segment, the dynamic linker executes the code in the. Init section to implement the initialization process that is unique to the shared object, such as the construction of a C + + global/static object in a shared object.

Dynamic Link also has a more flexible mode of module loading, called explicit run-time link, that is, let the program itself at run time to control the loading of the specified module, and can be unloaded when the module is not needed, such a shared object is often referred to as the dynamic loading library, can be used to implement such as plug-ins, drivers and other functions. There is no difference between a dynamic library and a generic shared object, except that the shared object is a dynamic linker that is responsible for loading and linking before the program starts, and that the process is transparent to the program itself, while the loading of the dynamic library is through a series of APIs provided by the dynamic linker. Specifically, there are 4 functions: Open the Dynamic Library (dlopen), find Symbol (DLSYM), error handling (DLERROR), and close the dynamic Library (dlclose), which the program can use to manipulate the dynamic library.

Dynamic links are about 1%-5% slower than static links in performance. There are two reasons to affect the performance of dynamic link, one is that when the program starts execution, the dynamic linker will do one link work, slowing down the program's startup speed, and the other is that the symbol reference outside the module needs to be accessed indirectly through got.

Compiling, linking, and loading under Linux