Build a testable Go Web Application

Build a testable Go Web Application

Almost every programmer agrees that testing is important, but testing in a variety of ways can help write testers. They may run slowly and may use repeated code. Too many tests at a time may make it difficult to locate the root cause of the test failure.

In this article, we will discuss how to design the Sourcegraph unit test to make it simple to write, easy to maintain, fast to run, and accessible to others. We hope that some of the models mentioned here will help others who write Go web apps. We also welcome suggestions on our testing methods. Before starting the test, let's take a look at our framework overview.

Framework

Like other web apps, our website has three layers:

• Web Front-end is used to serve HTML;

• Http api is used to return JSON;

• Data storage, run SQL queries on the database, and return the Go struct or slice.

When a user requests a Sourcegraph page, the front-end receives an HTTP page request and initiates a series of HTTP requests to the API server. Then, the API server starts querying the data storage. The data storage returns the data to the API server and then encodes it into JSON format and returns it to the web Front-End Server, the front-end uses the Go html/template package to display and format the data into HTML.

The frame chart is as follows: For more details, see recap of our Google I/O talk about building a large-scale code search engine in Go .)

 

Test v0

When we first started building Sourcegraph, we wrote the test in the easiest way to run. Each test enters the database and initiates an http get request to the test API endpoint. The test parses the HTTP returned content and compares it with the expected data. A typical v0 test is as follows:

 
 

  
  
func TestListRepositories(t *testing.T) {  
  
  
  tests := []struct { url string; insert []interface{}; want []*Repo }{  
  
  
    {"/repos", []*Repo{{Name: "foo"}}, []*Repo{{Name: "foo"}}},  
  
  
    {"/repos?lang=Go", []*Repo{{Lang: "Python"}}, nil},  
  
  
    {"/repos?lang=Go", []*Repo{{Lang: "Go"}}, []*Repo{{Lang: "Go"}}},  
  
  
  }  
  
  
  db.Connect()  
  
  
  s := http.NewServeMux()  
  
  
  s.Handle("/", router)  
  
  
  for _, test := range tests {  
  
  
    func() {  
  
  
      req, _ := http.NewRequest("GET", test.url, nil)  
  
  
      tx, _ := db.DB.DbMap.Begin()  
  
  
      defer tx.Rollback()  
  
  
      tx.Insert(test.data...)  
  
  
      rw := httptest.NewRecorder()  
  
  
      rw.Body = new(bytes.Buffer)  
  
  
      s.ServeHTTP(rw, req)  
  
  
      var got []*Repo  
  
  
      json.NewDecoder(rw.Body).Decode(&got)  
  
  
      if !reflect.DeepEqual(got, want) {  
  
  
        t.Errorf("%s: got %v, want %v", test.url, got, test.want)  
  
  
      }  
  
  
    }()  
  
  
  }  
  
  
} 
 
 

It was easy to write and test at the beginning, but it became painful as the app evolved. Over time, we added new features. More features lead to more tests, longer running time, and longer our dev cycle. More features also need to be changed and new URLs need to be added. Currently there are about 75 URLs), most of which are quite complicated. Each layer of Sourcegraph also becomes more complicated, so we want to test it independently of other layers.

We encountered some problems during the test:

1. Testing is slow because they want to interact with the actual database-insert test cases, initiate queries, and roll back each test transaction. Each test runs for about 100 milliseconds. As we add more tests.

2. Testing is difficult to refactor. The test uses a string to write the HTTP path and query parameters, which means that if we want to change a URL path or query the parameter set, we have to manually update the URL in the test. This pain will increase as the complexity and number of URL routes increase.

3. There are a large number of scattered and fragile sample codes. Install each test to ensure that the database runs normally and has the correct data. This code is used repeatedly in multiple cases, but the difference is sufficient to introduce bugs in the installation code. We found that we spent a lot of time debugging our tests instead of the actual app code.

4. It is difficult to diagnose a test failure. As the app becomes more complex, it is difficult to diagnose the root cause of test failure because each test accesses three application layers. Our tests are more like integration tests than unit tests.

Finally, we need to develop a public release API client. We want to make the API easy to be imitated so that our API users can also write test code.

Advanced testing objectives:

As our app evolves, we are aware of the need for tests that meet these high requirements:

• Clear objectives:We need to test each layer of the app separately.

• Comprehensive: All three layers of our app will be tested.

• Fast: The test must run very quickly, meaning no database interaction is performed.

• DRY: Although each layer of our apps is different, they share many common data structures. This is required for testing to eliminate duplicate sample code.

• Easy to imitate: External API users should also be able to use our internal test mode. Projects built on the basis of our API should be able to easily write good tests. After all, our web Front-end is not unique-it is just another API user.

How do we recreate the test

Well-written, maintainable testing and well-maintained application code are inseparable. Refactoring the application code allows us to greatly improve our test code, which is the step for us to improve the test.

1. Build a Go http api Client

The first step in simplifying testing is to use Go to write a high-quality client for our APIs. Previously, our website was an AngularJS app, but because we mainly serve static content, we decided to move the front-end HTML generation to the server. After that, our new front-end can use the Go API client to communicate with the API server. Our client go-sourcegraph is open-source, and the go-github library has a huge impact on it. The client code, especially the endpoint code used to obtain the repository data) is as follows:

 
 

  
  
func NewClient() *Client {  
  
  
  c := &Client{BaseURL:DefaultBaseURL}  
  
  
  c.Repositories = &repoService{c}  
  
  
  return c  
  
  
}  
  
  
   
  
  
type repoService struct{ c *Client }  
  
  
   
  
  
func (c *repoService) Get(name string) (*Repo, error) {  
  
  
    resp, err := http.Get(fmt.Sprintf("%s/api/repos/%s", c.BaseURL, name))  
  
  
    if err != nil {  
  
  
        return nil, err  
  
  
    }  
  
  
    defer resp.Body.Close()  
  
  
    var repo Repo  
  
  
    return &repo, json.NewDecoder(resp.Body).Decode(&repo)  
  
  
} 
 
 

In the past, our v0 API test killed a large number of URL paths and constructed HTTP requests in ad-hoc mode. Now they can use this API client to build and initiate requests.

2. Unified http api client and data warehouse Interfaces

Next, we will unify HTTP APIs and data warehouse interfaces. Previously, our API http. Handlers directly initiated SQL queries. Now our API http. Handlers only needs to parse http. Request and then call our data warehouse. The data warehouse and the http api client implement the same interface.

Using the above http api client (* repoService). Get method, we now have (* repoStore). Get:

 
 

  
  
func NewDatastore(dbh modl.SqlExecutor) *Datastore {  
  
  
  s := &Datastore{dbh: dbh}  
  
  
  s.Repositories = &repoStore{s}  
  
  
  return s  
  
  
}  
  
  
   
  
  
type repoStore struct{ *Datastore }  
  
  
   
  
  
func (s *repoStore) Get(name string) (*Repo, error) {  
  
  
    var repo *Repo  
  
  
    return repo, s.db.Select(&repo, "SELECT * FROM repo WHERE name=$1", name)  
  
  
} 
 
 

These interfaces are used to describe the behavior of our web app in one place, making it easier to understand and reason. In addition, we can reuse the same data type and parameter structure in the API client and data warehouse.

3. Centralized URL path Definition

Previously, we had to redefine the URL path at multiple layers of the application. In the API client, our code is as follows:

 
 

  
  
resp, err := http.Get(fmt.Sprintf("%s/api/repos/%s", c.BaseURL, name)) 
 
 

This method can easily cause errors, because we have more than 75 path definitions, and many of them are complicated. The centralized URL path definition means that the path is reconstructed in a new package independently from the API server. The path package defines the path.

 
 

  
  
const RepoGetRoute = "repo" 
  
  
   
  
  
func NewAPIRouter() *mux.Router {  
  
  
    m := mux.NewRouter()  
  
  
    // define the routes  
  
  
    m.Path("/api/repos/{Name:.*}").Name(RepoGetRoute)  
  
  
    return m  
  
  
}  
  
  
   
  
  
while the http.Handlers were actually mounted in the API server package:  
  
  
   
  
  
func init() {  
  
  
    m := NewAPIRouter()  
  
  
    // mount handlers  
  
  
    m.Get(RepoGetRoute).HandlerFunc(handleRepoGet)  
  
  
    http.Handle("/api/", m)  
  
  
} 
 
 

Http. Handlers is actually mounted in the API server package:

 
 

  
  
func init() {  
  
  
    m := NewAPIRouter()  
  
  
    // mount handlers  
  
  
    m.Get(RepoGetRoute).HandlerFunc(handleRepoGet)  
  
  
    http.Handle("/api/", m)  
  
  
} 
 
 

Now we can use the path package in the API client to generate URLs, instead of writing them to death. (* RepoService). Get method is as follows:

 
 

  
  
var apiRouter = NewAPIRouter()  
  
  
   
  
  
func (s *repoService) Get(name string) (*Repo, error) {  
  
  
    url, _ := apiRouter.Get(RepoGetRoute).URL("name", name)  
  
  
    resp, err := http.Get(s.baseURL + url.String())  
  
  
    if err != nil {  
  
  
        return nil, err  
  
  
    }  
  
  
    defer resp.Body.Close()  
  
  
   
  
  
    var repo []Repo  
  
  
    return repo, json.NewDecoder(resp.Body).Decode(&repo)  
  
  
} 
 
 

4. Create a copy of an ununified Interface

In our v0 test, we also tested path, HTTP processing, SQL generation, and DB query. Failure is difficult to diagnose, and testing is also slow.

Now, we have independent tests on each layer and have imitated the functions of the adjacent layer. Because each layer of the application implements the same interface, we can use the same imitation interface in all the three layers.

The Imitation implementation is a simple simulated function structure, which can be specified in each test:

 
 

  
  
type MockRepoService struct {  
  
  
    Get_ func(name string) (*Repo, error)  
  
  
}  
  
  
   
  
  
var _ RepoInterface = MockRepoService{}  
  
  
   
  
  
func (s MockRepoService) Get(name string) (*Repo, error) {  
  
  
    if s.Get_ == nil {  
  
  
        return nil, nil  
  
  
    }  
  
  
    return s.Get_(name)  
  
  
}  
  
  
   
  
  
func NewMockClient() *Client { return &Client{&MockRepoService{}} } 
 
 

The following describes the application in the test. We mimic the RepoService of the data warehouse and use the http api client to test the API http. Handler. (This Code uses all of the above methods .)

 
 

  
  
func TestRepoGet(t *testing.T) {  
  
  
   setup()  
  
  
   defer teardown()  
  
  
   
  
  
   var fetchedRepo bool  
  
  
   mockDatastore.Repo.(*MockRepoService).Get_ = func(name string) (*Repo, error) {  
  
  
       if name != "foo" {  
  
  
           t.Errorf("want Get %q, got %q", "foo", repo.URI)  
  
  
       }  
  
  
       fetchedRepo = true 
  
  
       return &Repo{name}, nil  
  
  
   }  
  
  
   
  
  
   repo, err := mockAPIClient.Repositories.Get("foo")  
  
  
   if err != nil { t.Fatal(err) }  
  
  
   
  
  
   if !fetchedRepo { t.Errorf("!fetchedRepo") }  
  
  
} 
 
 

Review of advanced testing objectives

With the above model, we achieved the test goal. Our code is:

• Clear objectives: One test layer.

• Comprehensive: All three application layers are tested.

• Fast: The test runs fast.

• DRY: We have combined three common interfaces at the application layer and reused them in application code and testing.

• Easy to imitate: One copy can be used in all three application layers. You can also use external APIs to test libraries built based on Sourcegraph.

The story about how to rebuild and improve the Sourcegraph test is complete. These models and examples run well in our environment. We hope these models and examples can also help others in the Go community, obviously, they are not correct in every scenario. We are sure there is room for improvement. We are constantly trying to improve our ways of doing things, so we are happy to hear your suggestions and feedback-let's talk about your experience in writing tests with Go!

From: http://www.oschina.net/translate/building-a-testable-webapp