Select the library where Go embeds static resources

Original: "Choosing A Library to Embed Static Assets in Go"

Author: Tim Shannon

Translator by: This article introduces several methods of Go embedding static files, then tells about the trouble they encountered during the use of a certain library, and finally compares the characteristics of different libraries. For developers who want to build static resources into executables, it can be useful. With the consent of the original author, I began the translation work.

Background

One of the often touted features of Go is that go apps are easy to deploy because the go-to-write program is statically compiled. But when you run a Web app, the advantage is basically gone if you need to manage the path and permissions of a series of files.

The workaround is to compile the necessary files into the application binaries. In Go, you can use byte slices to store byte content in a file in the form of a string literals.

 fileData := []byte("\x1f\x8b\ ... \x0f\x00\x00")

But a few of the biggest drawbacks of this approach are:

• Larger binary files
  • For my current project, the Lex Library, the executable file size is 20MB and embedded 21MB before the static file is embedded.
• Longer compile time
  • The latest Go compiler caching mechanism can effectively reduce build time.
• More memory at compile time
  • If you are developing with a small memory device, this will affect you. But personally, I don't have to worry about it.

You need to make a choice between development efficiency and operational management time. If your app audience is the general public (or the geek who has a personal online app), it's worth the pros and cons.

Embedding mode selection

The first one to handle the Go embedded static files, or the first truly well-known, should be Jteeuwen's go-bindata. This is a command-line application, you enter a file path that generates a file that contains a static file .go .

However, the author Jteeuwen seems to have left the planet and deleted everything on GitHub in this warehouse. Fortunately, his code is open source and is widely forked in the community. You can find several well-maintained fork in GitHub. The fork I chose at first was Shulhan, but then I chose another option, for the following reasons.

More details about the Jteewen project are here: HTTPS://GITHUB.COM/JTEEUWEN/GO-BINDATA/ISSUES/5

Alternative scenarios

Since Jteewen's library has a lot of alternatives. Here is an incomplete list of my search and collation:

• Vfsgen-https://github.com/shurcool/vfsgen
• Go.rice-https://github.com/geertjohan/go.rice
• Statik-https://github.com/rakyll/statik
• Esc-https://github.com/mjibson/esc
• Go-embed-https://github.com/pyros2097/go-embed
• Go-resources-https://github.com/omeid/go-resources
• Packr-https://github.com/gobuffalo/packr
• Statics-https://github.com/go-playground/statics
• Templify-https://github.com/wlbr/templify
• Gnoso/go-bindata-https://github.com/gnoso/go-bindata
• Shulhan/go-bindata-https://github.com/shulhan/go-bindata
• fileb0x-https://github.com/unnoted/fileb0x
• Gobundle-https://github.com/alecthomas/gobundle
• Parcello-https://github.com/phogolabs/parcello

The purpose of this article is to help you understand the differences in these libraries and to help you decide which features to consider when choosing one of the libraries above.

Chaff

With so many options, it makes a big head when deciding which one is best for your needs. Your program may not be the same as mine, but if it's a WEB application, there's a lot to learn from it. If you need to make the same choice, the comparisons below will be useful.

Judging criteria

Compression

As mentioned above, one drawback of embedding static files is that the executable file size is increased. You can use a library to compress the static files before embedding them, which can reduce some space. It also takes a bit of effort to decompress, but this is usually worthwhile because the memory footprint of the build is reduced in addition to space savings. In addition, Web page static file compression rates are high (compared to pictures), which is why most browsers support gzip compression. This introduces the next standard.

Optional decompression

If you have a static file with gzip compression in your executable file, and you want to make these files available to the client in gzip format, why not send these compressed files directly? The ideal library should support an option that allows you to set the file to accept compressed files directly at runtime without having to unzip them first.

Loading from the local file system

When you develop a WEB application, anything that adds time or difficulty should be avoided when you change the code and see the change. If you have to rebuild the Go program to statically embed the file every time you modify the CSS or HTML code, you will soon (will give up and) choose another scheme.

The ideal library should allow us to easily switch between programs built in development, quickly load static files locally, and in production, static files are embedded in executable programs and ready to be released.

Reproducible builds

This rule surprises me, and I didn't think about it at first when I started to develop the Lex Library. As mentioned earlier, the first library I chose was the go-bindata of Shulhan Fork. I chose it mainly because I was familiar with the original Go-bindata library, and this fork seemed to be well maintained.

It was a great library to use, but unexpectedly, my continuous integration (CI) build started to fail. Like every developer who has failed the test, I think about what was changed . I immediately check the latest submissions and try to figure out why these changes have caused the template processing to fail. After a moment of confusion, I did not find the reason, so I rerun the test suite, which was based on the last successful build of the code, but also failed. This means that the change is in the environment, not my code. But that raises more questions.

I ran the continuous integration test in a Docker container. The test environment should be self-wrapped, brand new, and reproducible. But my assumptions are not correct. When I look at Dockerfile it, I find the problem:

RUN go get -u github.com/shuLhan/go-bindata/...

Library Go-bindata has a small update, which affects my process of transmitting static file paths. All of a sudden, my embedded file path is different from what I expected. This may be a strange factor, such as go get always gets the default branch. Finally, it boils down to the simple fact that the module version of the code generated in the program has not been identified and has been tracked by my git repository. If the external code that the program relies on changes, it will unknowingly cause my build or test to fail.

One solution is to save a precompiled go-bindata executable to my Git project, but:

1. It is generally not appropriate to store binaries in a git repository.
2. If a dependent library fixes the vulnerability, I need to manually update go-bindata every time.
3. If you change a platform, it will make building difficult.

In addition, I found a library that does not require a separate binary file, and is completely dependent on the code version, which can be managed with vendor. This means there is a library that supports go-generate, but the difference is that it does not need to rely on external programs to run.

Additional standards

You may have different needs above, so in my comparison table below, there are some additional criteria that may be useful to you.

Configuration file

If you have a large number of different directories and files that need to be managed, use configuration files and put them in your source code, it's much easier to manage.

http. FileSystem interface

Implement HTTP. The FileSystem interface library makes embedding files easier to handle.

GitHub over 200 stars

This is a bit arbitrary, and the number of stars is not necessarily a way to measure the quality of code. However, the number of stars can explain the activity of the library, at least to indicate whether it is a multi-referenced library. This in turn also shows that many people are testing it, and/or feedback questions. I chose the library, just over the number of stars, pay attention to.

Contrast

Library	Compression	Optional Decompression	Local file System	go Generate	no executable file available	configuration file	http. FS	more than 200 stars
Vfsgen	✓	✓	✓*	✓	✓	✗	✓	✓
Go.rice	✓	✗	✓	✗	✗	✗	✓	✓
Statik	✓	✗	✗	✗	✗	✗	✓	✓
Esc	✓	✗	✓	✓	✗	✗	✓	✓
Go-embed	✓	✗	✓*	✗	✗	✗	✗	✗
Go-resources	✗	✗	✓*	✗	✗	✗	✓	✗
Packr	✓	✗	✓	✗	✗	✗	✓	✓
Statics	✓	✗	✓	✓	✗	✗	✓	✗
Templify	✗	✗	✓	✓	✗	✗	✗	✗
Gnoso/go-bindata	✓	✗	✗	✗	✗	✗	✗	✓
Shulhan/go-bindata	✓	✗	✓	✗	✗	✗	✗	✗
fileb0x	✓	✓	✓	✓	✗	✓	✓	✓
Gobundle	✓	✗	✗	✗	✗	✗	✗	✗
Parcello	✓	✗	✓	✓	✗	✗	✓	✓

* Additional code Required

My experience with these libraries is that they differ in both writing programs and deployment, to see the corresponding project README and documentation. If you see the inaccuracies in the table above, please let me know in the comments and I will update.

My choice.

Through the comparison table above, it is clear why I end up using Vfsgen, and I highly recommend it, especially if you need a reproducible build. A little bit less is fileb0x, which requires a standalone executable file to use go generate .