Video Note: Build mode for Go-David Crawshaw

This is a creation in Article, where the information may have evolved or changed.

• Video information
• What is Build Mode?
• Eight Build Mode of Go
  • exe(Static compilation)
  • EXE (with libc)
  • EXE (dynamic link libc and non-Go code)
  • pie-Position Independent executables
  • c-archiveC's Static link library
  • c-sharedC's dynamic link library
    • Why do I need dynamic links?
  • sharedDynamic link library for Go
  • pluginGo Plug-in
• Advantages and Disadvantages
• Future

Video Info #

Go Build Modes
by David Crawshaw, Google
At Gophercon 2017

Https://www.youtube.com/watch?v=x-LhC-J2Vbk

What is Build Mode? #

Build Mode is used to instruct the compiler how to create an executable binary file. The more you execute it, the more you can get the Go program to run in more places.

Go eight Build Mode #

• exe(Static compilation)
• exe(Dynamic link libc )
• exe(Dynamic links libc and non-Go codes)
• pieAddress-Independent executable file (security feature)
• c-archiveC's Static link library
• c-sharedC's dynamic link library
• sharedDynamic link library for Go
• pluginGo Plug-in

exe(Static compilation) #

This is everyone's favorite, all the code is built into an executable file.

1	Cgo_enabled=0 Go Build hello.go

This is the way you like to build with Go. All dependencies are built into a binary file, without any external dependencies, and the executable file is called directly and the syscall kernel communicates.

This CGO_ENABLED=0 is used to constrain the use CGO of no part, so that it does not depend on such libc libraries.

EXE (with libc) #

Most of these executables are statically compiled, except for the use of libc dynamic-link libraries, such as some net package operations, such as DNS queries, os/user user name queries, and so on, which use the system-provided libc dynamic link libraries.

The benefit is that system-specific implementations can be leveraged to ensure that the behavior is consistent with the system.

EXE (dynamic link libc and non-Go code) #

When the program compiles, all the go code is naturally compiled into an object file, and all non-go code can be compiled by its compiler (such as C, Fortran, etc.) into an object file, and these non-go code can be cgo called.

When the program is connected, these non-Go code can choose to be compiled into the final binary file, or you can choose the dynamic link to load at run time.

pie-Position Independent Executables #

This is the form of building a run-address-independent binary executable, which is a security feature that allows the executable file to be loaded with different addresses each time it loads in an operating system that supports PIE. Skip-the-hop attacks that avoid known addresses.

This is basically the same way as EXE, which may become the default in the future.

c-archiveC's Static link library #

Starting here, it's different from the previous build executable. This is a library that is built for C program calls. More precisely, here is the Go program to build a archive ( .a ) file, so that the class C program can statically link the .a file, and invoke the code.

• hello.go

1234567891011

Package mainimport"FMT"import"C"funcmain () {}//export HellofuncHello() {fmt. Println ("Hello, World")}

Note here that //export Hello this is the Convention, all the functions that need to be exported to C the call must be added with a comment, otherwise the header file required to generate C will not be built.

Then we build this hello.go file:

1	Go build-buildmode=c-archive hello.go

After the build, two files are generated, one is a static library file hello.a and the other is the header file of C hello.h .

12	HELLO.A: Current ar archive random libraryhello.h: C program text, ASCII text

In the generated hello.h header file, we can see the Hello() definition of Go function:

12345678910

#ifdef __cplusplusextern"C" {#endifextern voidHello(); #ifdef __cplusplus} #endif

Then we can refer to the hello.c header file in and use the Go compiled static library:

123456

#include"Hello.h"intmain(void){ Hello (); return 0;}

Then, build the C program:

1	CC HELLO.A Hello.c-o Hello

Final execution:

12	$./hellohello, world.

c-sharedC's Dynamic link library #

Unlike the previous example, this will create a dynamic-link library (Unix:/windows) with the Go code .so and .dll then dynamically load the run with the C language program.

The code for Go and C is the same as above, but the build process is different:

1	Go build-buildmode=c-shared-o hello.so hello.go

Here we use -buildmode=c-shared to build a dynamic link library supported by C.

Note: It is important to note that here, -o hello.so I am different from the speaker, if the output file name is not specified, then the default is used hello as the file name, resulting in subsequent operations cannot find the hello.so file.

This time also generated two files, one is hello.so , one is hello.h :

12	Hello.h: C program text, ASCII Texthello.so:mach-o 64-bit dynamically linked shared library x86_64

Then, compile the corresponding C program:

1	CC hello.c Hello.so-o Hello

If you compare the sizes of the binary c-archive executables in the examples and c-shared examples hello , you will find that c-shared the examples are hello much smaller:

12345

# c-archive-rwxr-xr-x 1 taowang staff 1.5M 3 Oct 17:51 Hello# c-shared-rwxr-xr-x 1 Taowang staff 8.2K 3 Oct 19:17 Hello

This is because the former, the code of Go statically compiled into the C program, and the latter, is a dynamic link, C's executable file does not contain the code we wrote Go, all this part of the function is in the dynamic link library hello.so .

1	-rw-r--r-- 1 Taowang staff 2.2M 3 Oct 19:17 hello.so

Therefore, when executing, we hello need hello.so this dynamic link library In addition to this binary executable file. If the default LD_LIBRARY_PATH contains the current directory and hello.so is in the current directory, you can directly:

12	$./hellohello, world.

Otherwise, if the hint is not found hello.so , such as:

1	Dyld:library not loaded:hello.so

You can specify the LD_LIBRARY_PATH variables manually and tell the operating system where to look for the dynamic link library:

1234567

# on Linux$ ld_library_path=. ./hellohello, World. # on MacOS$ dyld_library_path=. ./hellohello, World.

Why do I need dynamic links? #

From the start to use go we have repeatedly heard people say Go static link how convenient, so, then why do we need dynamic link?

Because dynamic links can be required at run time, the program decides to load, or can be uninstalled when not needed, which can save memory resources.

123456789101112131415

#include <dlfcn.h> #include < Stdio.h> int main (void ) {void * lib = dlopen (, 0 ); void (*FN) () = Dlsym (lib, ); if (!FN) {fprintf (stderr , "no fn:%s\n" , Dlerror ()); return 1 ; } //Calls Hello (); FN (); return 0 ;}

Here we use dlopen() to load the library and then use dlsym() it to load the symbol (function) into a function pointer, and then we call the function pointer fn() .

sharedGo's Dynamic link library #

sharedPatterns are c-shared similar to building a dynamic-link library for loading at run time. Instead of building a shared dynamic-link Library of C, the dynamic-link library is built specifically for Go executables.

Mode is not currently supported under MacOS shared .

This is still hello.go , but slightly different.

1234567

Package mainimport"FMT"funcmain() { Fmt. Println ("Hello, world!")}

Here is a standalone file, one main() that executes after printing Hello, World . We can exe build it in the same pattern as before and execute it. But this time we build it in a different way.

12	Go install-buildmode=shared stdgo build-linkshared hello.go

Here we first build and install the Go Standard library and std $GOPATH/pkg then use it -linkshared to build hello.go .

The execution result is the same as before, but if you look closely at the resulting file, you will find it very different from the previous one.

12	$ ls-l hello-rwxr-xr-x 1 root root 16032 Oct 3 13:27 Hello

You can see that this Hello world program is only more than 10 KB in size. For C programmers, this is not surprising, because it should be so big ah. But for Go programmers, this is very strange, because generally do not have to 7~8mb it?

The reason for this is that the dynamic link library is used, and all the standard library parts are called dynamically linked, and the built binary executable contains only the program part. The reason why the C program builds the Hello world is also because of the dynamic link.

If we look at the library called by the program, we can see the specifics:

1234567

$ ldd Hello linux-vdso.so.1 (0x00007ffed3d4e000) libstd.so =/usr/local/go/pkg/linux_amd64_ Dynlink/libstd.so (0x00007f608c409000) libc.so.6 =/lib/x86_64-linux-gnu/libc.so.6 (0x00007f608c06a000) libdl.so.2 =/lib/x86_64-linux-gnu/libdl.so.2 (0x00007f608be66000) libpthread.so.0 =/lib/x86_64- linux-gnu/libpthread.so.0 (0x00007f608bc49000) /lib64/ld-linux-x86-64.so.2 (0x00007f608e866000)

If we go further libstd.so , we will see a huge dynamic link library, which is the standard library for go:

1	-rw-r--r--1 root root 37M Oct 3 13:27/usr/local/go/pkg/linux_amd64_dynlink/libstd.so

Of course, to use this mode requires a lot of preparation, all the dynamic link libraries need to be in the specified location, the version must be compatible and so on, so we generally do not use this pattern.

pluginGo Plugin #

Plug-in form and c-shared , shared similar, is to build a dynamic link library, and the shared same, this is to build a Go dedicated dynamic link library, and the shared difference is that the dynamic link library is not loaded at the start of the program, but in the program to decide when to load and release.

This is a new thing, so it may not be possible to use?......, of course, if you use the right, it should still be used.

We create a plugin that myplugin.go :

1234567

Package mainimport"FMT"funcHello() { fmt. Println ("Hello, world!")}

As you can see, this is similar to the nature of the original static link library. But the difference is that there is neither, nor is there import "C" //export Hello , and there is no func main() . Because this is not necessary, we are going to call go code, so a lot of things are saved.

The calling code writes this:

123456789101112131415161718

package mainimport "plugin" func main () {//load myplugin library p, err: = plugin. Open ( "myplugin.so" ) if err! = nil {log. Fatal (Err)}//get Hello function fn, err: = P.lookup ( "Hello" ) if err! = nil {log. Fatal (Err)}//call function fn. (func Span class= "params" > () ) () }

Can be seen, this logically, and hello-dyn.c very similar. Kind plugin.Open() dlopen() of like p.Lookup() dlsym() . This is actually the case, when the underlying implementation is called the two functions.

Note here that fn.(func())() the p.Lookup() return is one interface{} , so this needs to be transformed into a specific function type.

Build with the following command:

12	Go build-buildmode=plugin Myplugin.gogo Build Runplugin.go

The former generates one myplugin.so , which generates the caller runplugin .

12	-rw-r--r--1 root root 3.8M Oct 3 13:58 myplugin.so-rwxr-xr-x 1 root root 3.5M Oct 3 13:58 runplugin

Advantages and Disadvantages

• exe(Static compilation)
  • Pros:
    • Fully integrated, no dependencies required
    • Ideal for ultra-small container environments
    • Easy to cross different Linux distributions
• exe(Dynamic link libc )
  • Pros:
    • You can take advantage of system functions, such as DNS queries.
    • You can libc use the system configuration directly.
  • Cons:
    • Execution environment that relies on user space
• exe(Dynamic links libc and non-Go codes)
  • Pros:
    • You can use non-go code directly in the GO program
    • Easy and old system integration
  • Cons:
    • Build becomes more complex
    • C not Go
    • More prone to problems.
      • All Go may be problematic places
      • Where all C could go wrong.
      • Where the communication between Go <-> C could be problematic
• pieAddress-Independent executable file (security feature)
  • Pros:
    • and the exe same
    • Make the system more difficult to attack
  • Cons:
    • The binary is bigger (bug, will is fixed)
    • There will be a ~1% loss of performance.
• c-archiveC's Static link library
  • Pros:
    • Allows go integration into existing C programs
    • In fact, that's how Go works on IOS.
    • Ideal for building existing non-Go environments
  • Cons:
    • Cross-language calls can be tricky
• c-sharedC's dynamic link library
  • Pros:
    • Easier go integration into existing C programs
    • Can be loaded at run time
    • This is how Go works under Android (Java System.load() )
  • Cons:
    • Cross-language calls can be tricky
    • Consider the Android environment, which may have a bigger problem area:
      • Where Go might be wrong
      • C where there may be a problem
      • Where Java may be problematic
      • All the places where communication between them could be problematic ...?
• sharedDynamic link library for Go
  • Pros:
    • Multiple executables can share a dynamic-link library, reducing the total volume of the system.
    • General operating system vendors will be more favored in this way, can make the overall system of volume reduction.
      • In fact, this is the way Canonical (Ubuntu) Force is implemented
    • Can reduce the system volume, but now storage space is not a problem
    • Can reduce memory, you can share dynamic-link library code in memory (if the dynamic-link library's loader is smart enough)
  • Cons:
    • Dependency management, and publishing are very rare
      • However, the general operating system vendors already have a mature publishing system
• pluginGo Plug-in
  • Pros:
    • The Go program loads other go programs at run time
    • Allows different parts to be built at different times for complex applications
  • Cons:
    • Build is complex, and deployment can be complex
    • If asked whether the speaker should use the plugin pattern, the answer is generally No.

Future

There are many other areas that need improvement.

• c-shared : Windows not currently supported (possible), MacOS (partially supported)
• shared : MacOS is currently not supported
• plugin : MacOS is not currently supported, Windows
• plugin : You may be able to remove the runtime from the plugin to get a smaller executable file