Go language random test tool Go-fuzz

This is a creation in Article, where the information may have evolved or changed.

When Go 1.5 was released, the former Intel Black belt level engineer, now Google engineer Dmitry Vyukov, also released the Go Language Random test tool go-fuzz. At the GOPHERCON2015 conference, Dmitry Vyukov highlighted go-fuzz in presentation, which is called "Go Dynamic Tools".

The Go-fuzz is a random test (testing) tool. A lot of people are unfamiliar with random testing, and I'm no exception. I've never used a similar test tool in Golang or other programming languages before I've contacted Go-fuzz (c + + developers can use Afl-fuzz). According to Wikipedia, random testing refers to the automatic or automatic provision of illegal, unintended, random data for the program, and monitors the program's crash, built-in assertions, memory leaks, and so forth under these input data. The study of random tests began in 1988 with Barton Miller, which has so far been supported by many theories, but is not involved, and interested, in-depth friends can follow the links in Wikipedia to learn by themselves.

Before we start go-fuzz, we need to recognize the location and significance of random testing:
* First, it is an important branch of software testing technology, and unit testing and other complementary;
* Followed by random testing is not what the silver bullet, it has its applicable range. Random tests are best suited for applications that handle complex input data, such as file format parsing, network protocol parsing, and human-computer interface entry.
* Finally, not all programming languages have similar tool support, Gopher Fortunately, Dmitry Vyukov brings us go-fuzz.

Then let's go back to Go-fuzz.

First, why Go-fuzz

Go-fuzz's eye-catching comes from Dmitry Vyukov's "amazing victories" after testing the Go standard library and other third-party open source libraries with Go-fuzz. Dmitry demonstrated these victories in his slide:

60 tests137 bugs in std lib (70 fixed)165 elsewhere (47 in gccgo, 30 in golang.org/x, 42 in freetype-go, protobuf, http2, bson)

The Go-fuzz of Dmitry Vyukov is actually designed and implemented on the basis of the aforementioned afl-fuzz logic. The difference is that when used, the Afl-fuzz fork a process for each input case, while Go-fuzz passes the data in input cases to a fuzz function:

func Fuzz(data []byte) int

This eliminates the need to restart the program repeatedly.

Go-fuzz further perfected the Go Development test toolset, and many of the first-line companies (such as CloudFlare) have started to use Go-fuzz to test their products and improve product quality.

Second, the principle

Dmitry in its slide the Go-fuzz workflow is summarized as follows:

 -> 生成随机数据 -> 输入给程序 -> 观察是否有crash -> 如果发现crash，则获益  之后开发者根据crash的结果，尝试fix bug，并  添加针对这个bug的单元测试case。

Once the Go-fuzz is running, it will be a infinite loop (a genetic algorithm), and the loop's pseudo-code is also given in slide:

Instrument program for code coverageCollect initial corpus of inputs  //收集初始输入数据语料(位于workdir的corpus目录下)for {    //从corpus中读取语料并随机变化    Randomly mutate an input from the corpus    //执行Fuzz，收集覆盖范围    Execute and collect coverage    //如果输入数据提供了新的coverage，则将该数据存入语料库(corpus)    If the input gives new coverage, add it to corpus}

The Go-fuzz internally implements a variety of mutation strategies for inputting data into the initial corpus:

* Insert/remove/duplicate/copy a random range of random bytes.* Bit flip.* Swap 2 bytes.* Set a byte to a random value.* Add/subtract from a byte/uint16/uint32/uint64 (le/be).* Replace a byte/uint16/uint32 with an interesting value (le/be).* Replace an ascii digit/number with another digit/number.* Splice another input.* Insert a part of another input.* Insert a string/int literal.* Replace with string/int literal.

Iii. Methods of Use

1, installation Go-fuzz

Using Go-fuzz requires the installation of two important tools: Go-fuzz-build and Go-fuzz, which can be installed by standard go get:

$ go get github.com/dvyukov/go-fuzz/go-fuzz$ go get github.com/dvyukov/go-fuzz/go-fuzz-build

For domestic users, because Go-fuzz does not use the vendor mechanism introduced by Go 1.5, some of the packages it relies on are outside the walls, and therefore may encounter some problems.

Go get automatically installs two tools to $goroot/bin or $gopath/bin, so you need to make sure that your PATH environment variable contains both paths.

2. Project organization with Fuzz test

Suppose our go package to be tested is named Foo, and the path is $gopath/src/github.com/bigwhite/fuzzexamples/foo. In order to apply go-fuzz, we typically create a fuzz.go source file under Foo with the following content template:

// +build gofuzzpackage foofunc Fuzz(data []byte) int {    ... ...}

Go-fuzz the source files with "+build gofuzz" directive and fuzz functions in them are searched when building a binary file for fuzz test execution. If the file does not appear under the Foo package, you will get an error log similar to the following when you execute Go-fuzz-build:

$go-fuzz-build github.com/bigwhite/fuzzexamples/foofailed to execute go build: exit status 2# go-fuzz-main/var/folders/2h/xr2tmnxx6qxc4w4w13m01fsh0000gn/T/go-fuzz-build641745751/src/go-fuzz-main/main.go:10: undefined: foo.Fuzz

There are times when there are a lot of features in the test package, a fuzz function is not enough, we can refer to Go-fuzz in the example directory organization to deal with:

github.com/bigwhite/fuzzexamples/foo/fuzztest]$tree.├── fuzz1│   ├── corpus│   ├── fuzz.go│   └── gen│       └── main.go└── fuzz2    ├── corpus    ├── fuzz.go    └── gen        └── main.go ... ...

The FUZZ1, fuzz2 .... FUZZN are each a go-fuzz unit, and if you want to apply Go-fuzz, you can do this as follows:

$ cd fuzz1$ go-fuzz-build github.com/bigwhite/fuzzexamples/foo/fuzztest/fuzz1$ go-fuzz -bin=./foo-fuzz.zip -workdir=./.. ...$ cd fuzz2$ go-fuzz-build github.com/bigwhite/fuzzexamples/foo/fuzztest/fuzz2$ go-fuzz -bin=./foo-fuzz.zip -workdir=./

There is a set of "fixed" directory combinations for each Go-fuzz unit:

├── fuzz1│   ├── corpus│   ├── fuzz.go│   └── gen│       └── main.go

Corpus is the directory where the input data corpus is stored and the initial corpus can be placed before the Go-fuzz executes;
Fuzz.go is the source file containing the fuzz function;
The Gen directory contains the Main.go code that generates the initial corpus manually.

In the following example, we will show the details.

3, Go-fuzz-build

Go-fuzz-build will build a Zip package (packagename-fuzz.zip) for Go-fuzz execution based on the Fuzz function, which contains three files with different uses:

-rw-r--r--   1 tony  staff  3902136 12 31  1979 cover.exe-rw-r--r--   1 tony  staff  3211816 12 31  1979 metadata-rw-r--r--   1 tony  staff  5031496 12 31  1979 sonar.exe

According to author slide, the functions of each binary program are as follows:
Cover.exe–coverage instrumented binary
Sonar.exe–sonar instrumented binary
Metadata–coverage and sonar metadata, int and string literals

But for users, we don't have to be overly concerned about them, donuts.

4, the implementation of Go-fuzz

Once the foo-fuzz.zip is generated, we can execute the fuzz test for FUZZ1.

$ cd fuzz1$ go-fuzz -bin=./foo-fuzz.zip -workdir=./2015/12/08 17:51:48 slaves: 4, corpus: 8 (1s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s2015/12/08 17:51:51 slaves: 4, corpus: 9 (2s ago), crashers: 0, restarts: 1/3851, execs: 11553 (1924/sec), cover: 143, uptime: 6s2015/12/08 17:51:54 slaves: 4, corpus: 9 (5s ago), crashers: 0, restarts: 1/3979, execs: 47756 (5305/sec), cover: 143, uptime: 9s... ...

If there is no initial corpus data in the corpus, then Go-fuzz will generate the relevant data to fuzz function, and use genetic algorithm to generate a new input corpus based on the corpus of Corpus. Go-fuzz authors suggest that the more corpus the corpus initially put in, the better, and enough diversity, so that the genetic algorithm based on these initial corpora will be more effective. Go-fuzz will persist some of the corpus into a file in corpus for the next restart use.

As I said earlier, Go-fuzz is a infinite loop, and the above tests need to be stopped manually. Go-fuzz will create two additional directories in Workdir: Crashers and suppressions. As the name implies, Crashers is the data that is stored in the code crash, including the input binary data of the case that caused the crash, the string form of the input data (xxx.quoted), and the output data (Xxx.output) based on this data. The stack trace information for crash is stored in the suppressions.

四、一个 Simple Example

GOCMPP is a go implementation of the CMPP Protocol Library, which intends to use unpack as the simplest Fuzz test demo.

Each protocol package in GOCMPP implements the Packer interface, where unpack is particularly suitable for fuzz test. Due to the large number of protocol packages, we have created a fuzztest directory under GOCMPP to store fuzz test code and separate the fuzz test for each protocol package into subdirectories:

github.com/bigwhite/gocmpp/fuzztest]$tree.├── fwd│   ├── corpus│   │   └── 0│   ├── fuzz.go│   └── gen│       └── main.go└── submit       ├── corpus       │   ├── 0       ├── fuzz.go       └── gen           └── main.go

Let's start with the gen/main.go of each fuzz test unit (such as FWD or submit), which is an executable program for generating the initial corpus, and we take submit/gen/main.go as an example:

Package Mainimport ("Github.com/dvyukov/go-fuzz/gen") func main () {data: = []byte{0x00, 0x00, 0x00, 0x17, 0 X00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x01, 0x01, 0x01, 0x74, 0x65, 0x73, 0x74, 0x00, 0x00, 0x00, 0x XX, 0x00, 0x00, 0x02, 0x31, 0x33, 0x35, 0x30, 0x30, 0x30, 0x30, 0x32, 0x36, 0x39, 0x36, 0x00, 0x00, 0x00, 0x XX, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x08, 0x39, 0x30, 0x30, 0x30, 0x30, 0x31, 0x30, 0x32, 0x31, 0x3 0, 0x00, 0x00, 0x00, 0x00, 0x31, 0x35, 0x31, 0x31, 0x30, 0x35, 0x31, 0x33, 0x31, 0x35, 0x35, 0x35, 0x31, 0x30, 0x31  , 0x2b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x39, 0x30, 0x30, 0x30, 0x30, 0x31, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,        0x00, 0x00, 0x01, 0x31, 0x33, 0x35, 0x30, 0x30, 0x30, 0x30, 0x32, 0x36, 0x39, 0x36, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,0x00, 0x00, 0x1e, 0x6d, 0x4b, 0x8b, 0xd5, 0x00, 0x67, 0x00, 0x6f, 0x00, 0x63, 0x00, 0x6d, 0x00, 0x70, 0x00, 0x70, 0 X00, 0x20, 0x00, 0x73, 0x00, 0x75, 0x00, 0x62, 0x00, 0x6d, 0x00, 0x69, 0x00, 0x74, 0x00, 0x00, 0x00, 0x00, 0x00, 0x XX, 0x00, 0x00,} gen. Emit (data, nil, True)}

In this main.go, we use the data from the unit test of the submit package as the initial corpus data for fuzz test, and output the data to a file through the Gen package provided by Go-fuzz:

$cd submit/gen$go run main.go -out ../corpus/$ll ../corpus/total 8drwxr-xr-x  3 tony  staff  102 12  7 22:00 ./drwxr-xr-x  5 tony  staff  170 12  7 21:42 ../-rw-r--r--  1 tony  staff  181 12  7 22:00 0

The program generates a file "0" under Corpus as the initial corpus for the submit fuzz test.

Next we look at Submit/fuzz.go:

// +build gofuzzpackage cmppfuzzimport (    "github.com/bigwhite/gocmpp")func Fuzz(data []byte) int {    p := &cmpp.Cmpp2SubmitReqPkt{}    if err := p.Unpack(data); err != nil {        return 0    }    return 1}

This is a "simplest" fuzz function implemented, according to the author of the Fuzz protocol, the return value of fuzz is important:

如果此次输入的数据在某种程度上是很有意义的，go-fuzz会给予这类输入更多的优先级，Fuzz应该返回1；如果明确这些输入绝对不能放入corpus，那让Fuzz返回-1；至于其他情况，返回0。

The next step is Go-fuzz-build and Go-fuzz, which is similar to the previous introduction:

$cd submit$go-fuzz-build github.com/bigwhite/gocmpp/fuzztest/submit$lscmppfuzz-fuzz.zip    corpus/            fuzz.go            gen/

Execute Go-fuzz under the Submit directory:

$go-fuzz -bin=./cmppfuzz-fuzz.zip -workdir=./2015/12/07 22:05:02 slaves: 4, corpus: 1 (3s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s2015/12/07 22:05:05 slaves: 4, corpus: 3 (0s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 32, uptime: 6s2015/12/07 22:05:08 slaves: 4, corpus: 7 (1s ago), crashers: 0, restarts: 1/5424, execs: 65098 (7231/sec), cover: 131, uptime: 9s2015/12/07 22:05:11 slaves: 4, corpus: 9 (0s ago), crashers: 0, restarts: 1/5424, execs: 65098 (5424/sec), cover: 146, uptime: 12s... ...2015/12/07 22:09:11 slaves: 4, corpus: 9 (4m0s ago), crashers: 0, restarts: 1/9860, execs: 4033002 (16002/sec), cover: 146, uptime: 4m12s^C2015/12/07 22:09:13 shutting down...

This test is very CPU consuming! For a little while, the fan of my Mac Air started to turn up. But my unpack function didn't find a problem in fuzz test, and the value behind Crashers was 0.

Go-fuzz currently does not seem to support the vendor mechanism, so if your package uses vendor like GOCMPP, you need to add a Go-fuzz-build "Go-fuzz" in front of go15vendorexperiment= and 0″ ( If you have previously opened Go15vendorexperiment), just like this:

$ GO15VENDOREXPERIMENT="0" go-fuzz-build github.com/bigwhite/gocmpp/fuzztest/submit

If you do not close vendor, you may get an error similar to the following:

can't find imported package golang.org/x/text/transform

Bigwhite. All rights reserved.