Translation GO 1.3 linker overhaul

This is a creation in Article, where the information may have evolved or changed.

Go team can always bring some surprises, about the GO linker, it seems to be in the 1.3 version of the overhaul.
———— Translation Divider Line ————

GO 1.3 linker overhaul

Russ Cox

November 2013

Summary

The linker is the slowest part when building and running a standard Go program. To solve this problem, we plan to split the linker into two parts. Some of them may be written in Go.

Background

The linker is always one of the slowest parts of the Plan 9 tool chain, and now it is the slowest part of the Go tool chain. Ken Thompson concludes with an overview of the toolchain:

The new compiler compiles quickly, loads slowly, and produces medium-quality target code. The compiler is related to portability and requires several weeks of work for different computers to build the corresponding compiler. For Plan 9, it is necessary to have several compilers that have specific functionality and use their own target format. For us, the compiler must be free to accompany Plan 9 release.

Two questions were brought up in the review. First, the compiler and the loader must be simplified. Plan 9 runs on a variety of processors, and these compilations are usually done in parallel. Unfortunately, all compilations must be completed before the load is made. The load is single-threaded. In this model, any changes to the compiled results for loading will significantly increase the actual time. The same is true for libraries that are often compiled and loaded. In the future, we may try to put some loading work into the compiler to complete.

This document was written in the early 90 's. Now is the future.

Proposal Planning

The current linker performs two separate tasks. First, it accompanies the positional table, translating the pseudo-instruction input stream into executable code and data blocks. It then removes the useless code, merges the other into a single mirror, re-handles the location, and generates some complete program data structures such as the run-time symbol table.

The first part can be decomposed into a library (liblink) so that it can be federated with the assembler and the compiler. Those 6a, 6c, 6g, and so on output of the target file can be output by Liblink, and contains executable code, data block and positioning table, which is the first intermediate product of the current linker.

The second part can be handled by the linker after removing the liblink. The rest of the program can read the new target file and complete the link. This linker has very little code, most of it is schema-independent. This makes it possible to merge it into a platform-independent program and then invoke it like "go tool ld". Even use Go to write it, making it easier to parallelize in large links. (See the section below to learn how to get started.) ）

In the beginning, we will focus on making the newly stripped part available for C code. Once all the modifications have been completed, go to explore.

To avoid affecting availability, the resulting target file will keep the extension already in place. 5,. 6,. 8. Perhaps in Go 1.3, the new linker can contain a transitional program called 5l, 6l, and 8l. These transition programs will be removed in Go 1.4.

Target file

The new split requires a new destination file format. The current target file contains a pseudo-instruction stream, but the new destination file will contain executable code and data blocks accompanying the locating table.

A natural question is whether you should use an existing target file format, such as ELF. First, we'll use a custom format. To build a run-time data structure like a run-time symbol table, you need to customize a GO linker, so you won't be able to reuse the standard ELF linker even with Elf target files. The Elf file also contains a large number of common semantics and ELF semantics that are required to exceed a GO custom linker. A custom, less generic, target file format makes it easier to build and use. On the other hand, ELF can be handled by standard tools such as readelf, objdump, etc. However, once the dust settles, when we know exactly what this format needs, it is worth confirming whether the use of ELF can satisfy the needs.

The details of the new target file are not yet complete. The remainder of this section lists some of the design ideas.

Obviously, the simpler the file, the better. In addition to some exceptions, the work of the linker should be done in the intermediate library. Might magically contain the stack partition code, which makes the target files dependent on the operating system, although they are already based on the OS-dependent Go code in the package. At the same time, the floating-point work of the software is also done in the intermediate library, so that the ARM target file is a specific goarm (currently nothing is goarm until the linker is running).
We need to make sure that the target file is available through mmap. This can reduce the I/O that is caused by replication. This requires modifying the Go runtime to generate a SIGSEGV for non-nil addresses, creating a panic instead of crashing.
A pure go package produces a target file once by the go compiler on a complete collection of Go code files. The target file is then encapsulated in a packaged file. We can tidy up this single target file is also a legitimate packaging file, then usually there is no need to encapsulate the steps.

Start

If the new GO linker is written in go, there is a startup question: How do I link to this linker? Here are two options.

The first scenario is to maintain a CL boot list. The first CL in the queue should contain the current linker, written by C. Each step is a CL that contains the new linker used to link the next one. The last binary result of the queue is available for download. The queue cannot be too long and can fit the milestone. For example, the GO 1.3 linker can be compiled by the Go 1.2 program, the GO 1.4 linker can be compiled by the Go 1.3 program, and so on. The queue's record ensures that it can be restarted if needed, but also provides a mechanism to deal with trust-dependent issues. Another way to start is to use GCCGO and then use it to compile the GO 1.3 linker.

The second scenario is that after writing a better linker with Go, the version of C is still maintained and the majority of the functions are equivalent. The version written in C only needs to maintain enough functionality to link this to the Go writing. It needs to load some target files, merge them, and then output the executable file. No CGO support, no external links, no shared libraries, no high performance. You just need some plain code (probably just thousands of lines of code) and don't need to be modified frequently. The C version will be built and used at Make.bash, but will not be installed. This scheme is easier for other developers who build from Go source.

It doesn't matter which option we choose, at least one option is available. If things go on, we can decide.