How go uses go to build itself

This is a creation in Article, where the information may have evolved or changed.

This post was based on a talk I gave to the Sydney Go Users group in mid April describing the Go build process.

frequently on mailing list or IRC channel there is requests for documentation on the details of the Go compiler, runtime and internals. Currently the canonical source of documentation about Go ' internals are the source, which I encourage everyone to read. Have said that, the go build process have been stable since the Go 1.0 release, so documenting it's here'll probably Rema In relevant for some time.

This post walks through the nine steps of the go build process, starting with the source and ending with a fully tested go Installation. For simplicity, all paths mentioned is relative to the root of the source checkout, $GOROOT/src .

For background your should also read installing Go from source on the golang.org website.

Step 1. All.bash

% CD $GOROOT/src%./all.bash

The first step is a bit anticlimactic as all.bash just calls, and the other shell scripts; make.bash run.bash If you ' re using a Windows or Plan 9, the process is the same, but the scripts end in .bat or .rc respectively. For the rest of this post, please substitute the "the" extension appropriate for your operating system.

Step 2. Make.bash

. ./make.bash--no-banner

make.bashis sourced from all.bash so, calls exit to would terminate the build process properly. make.bash has three main jobs, the FIR St job is to validate the environment Go are being compiled in is sane. The sanity checks has been built up through the last few years and generally try-to-avoid building with known broken tools, Or in environments where the build would fail.

Step 3. Cmd/dist

Gcc-o2-wall-werror-ggdb-o cmd/dist/dist-icmd/dist cmd/dist/*.c

Once The sanity checks is complete,make.bashCompilescmd/dist.cmd/distReplaces theMakefileBased system which existed before Go 1 and manages the small amounts of code generation inpkg/runtime.cmd/distis a C program which allows it to leverage the system C compiler and headers to handle most of the host platform detection Issues.cmd/distAlways detects your host ' s operating system and architecture,$GOHOSTOSand$GOHOSTARCH. These differ from any value of$GOOSand$GOARCHYou may have the set if is cross compiling. In fact, the Go build process is all building a cross compiler, but the cases of the host and target platform are the Same. Next,make.bashInvokescmd/distWith the bootstrap argument which compiles the supporting libraries,lib9,libbioandlibmach, used by the compiler suite and then the compilers themselves. These tools is also written in C and is compiled by the system C compiler.

echo "# Building compilers and Go bootstrap tool for Host, $GOHOSTOS/$GOHOSTARCH." Buildall= "-A" if ["$" = "--no-clean"]; Then buildall= "" Fi./cmd/dist/dist Bootstrap $buildall-V # builds Go_bootstrap

Using the compiler suite, then cmd/dist compiles a version go of the tool, go_bootstrap . go_bootstrapThe tool is isn't the full go tool and for example is pkg/net stubbed out which avoids a dependency on cgo . The list of directories containing packages or libraries be compiled, and their dependencies are encoded in the cmd/dist tool itself, so great care are taken to avoid introducing new build dependencies for cmd/go .

Step 4. Go_bootstrap

Now go_bootstrap It is built, the final stage of the was to use make.bash go_bootstrap to compile the complete Go standard library, including a Replacement version of the full go tool.

echo "# Building packages and commands for $GOOS/$GOARCH." $GOTOOLDIR "/go_bootstrap install-gcflags" $GO _gcflags "\    -ldflags" $GO _ldflags "-V STD

Step 5. Run.bash

Now that's complete make.bash , execution falls back all.bash to, which invokes run.bash . run.bash ' s job is to compile and test the standard library, the runtime, and the language test suite.

Bash Run.bash--no-rebuild

--no-rebuildthe flag is used because make.bash and run.bash can both invoke go install -a std , so to avoid duplicating the previous effort, --no-rebuild s Kips the second go install .

# allow All.bash to avoid double-build of Everythingrebuild=trueif ["$" = "--no-rebuild"]; Then Shiftelse Echo ' # Building packages and commands. ' Time go install-a-v STD Echofi

Step 6. Go test-a STD

Echo ' # Testing packages ' time go test std-short-timeout=$ (expr \* $timeout _scale) Secho

Next run.bash is to run the unit tests for all the packages in the standard library, which is written using the testing Packag E. Because code $GOPATH $GOROOT in and live with the same namespace, we cannot use as this go test ... would also test every package I n $GOPATH , so an alias, std is created to address the packages in the standard library. Because Some tests take a long time, or consume a lot of memory, some tests filter themselves with the -short flag.

Step 7. Runtime and CGO tests

run.bashthe next section of runs a set cgo of tests for platforms, runs a few benchmarks, and compiles miscel Laneous programs that ship with the Go distribution. Over time this list of miscellaneous programs have grown as it is found that time they were not included in the build proc ESS, they would inevitably break silently.

Step 8. Go Run test

(XCD: /testunset gomaxprocstime Go run run.go) | | Exit $?

The penultimate stage of run.bash invokes the compiler and runtime tests in the test folder directly under $GOROOT . These is tests of the low level details of the compiler and runtime itself. While the tests exercise the specification of the language, the test/bugs and test/fixedbugs sub directories capture unique tests for I Ssues which has been found and fixed. The test driver for all these tests is $GOROOT/test/run.go which are a small Go program this runs each .go file inside the test Directo Ry. Some .go files contain directives on the first line which instruct run.go to expect, for example, the program to fail, or To emit a certain output sequence.

Step 9. Go Tool API

 Echo ' # Checking API compatibility. ' Go tool api-c $GOROOT/api/go1.txt, $GOROOT/api/go1.1.txt \-next $GOROOT/api/n Ext.txt-except $GOROOT/api/except.txt 

The final step of the are to run.bash invoke the api tool. apithe tool ' s job is to enforce the Go 1 contract; The exported symbols, constants, functions, variables, types and met Hods that made up the Go 1 APIs when it shipped in 2012. For go 1 They is spelled out in api/go1.txt , and Go 1.1, api/go1.1.txt . An additional file, identifies the symbols, the additions to the standard api/next.txt library and runtime since Go 1 .1. Once Go 1.2 Ships, this file would become the contract for Go 1.2, and there would be a new next.txt . There is also a small file, except.txt and which contains exceptions to the Go 1 contract which has been approved. Additions to the file is not expected to be taken lightly.

Additional Tips and Tricks

You ' ve probably figured off that's make.bash useful for building Go without running the tests, and likewise, is run.bash useful For building and testing the Go runtime. This distinction is also useful as the former can being used when cross compiling Go, and the latter was useful if you are wor King on the standard library.

Update: Thanks to Russ Cox and Andrew Gerrand for their feedback and suggestions.

Related Posts:

1. How do does the go build command work?
2. What does go build build?
3. Using go test, build and install
4. Using Juju to build Gccgo