Go microservices Series-Part III-embedding databases and JSON

This is a creation in Article, where the information may have evolved or changed.

Part III: Go microservices-embed databases and JSON

In the third part, we let accountservice do something meaningful.

• Declares an account structure.
• Embedded in simple key-value storage, we can store the account structure inside.
• Serializes the struct into JSON and then serves the/accounts/{accountid} through the HTTP service.

Source

Source code location: Https://github.com/callistaen ....

Declaring the account structure body

A detailed description of the structure can be found in the relevant links section of the reference link.

1. Create a directory named model under our project root directory, Accountservice.
2. Create the Account.go file under the Model directory.

package modeltype Account struct {    Id string `json:"id"`    Name string `json:"name"`}

The account is abstracted into a struct containing the ID and name. The two attributes of the struct are capitalized, indicating that the global scope is declared visible (the initial capital of the identifier is public and the first lowercase package scope is visible).

In addition, the tag is used in the structure. These labels have special applications in Encoding/json and Encoding/xml.

Suppose we do not use tags when defining structs, and JSON for structs. The JSON key generated after Marshal uses the value corresponding to the struct field name.

For example:

type Account struct {    Id string    Name string}var account = Account{    Id: 10000,    Name: "admin",}

After converting to JSON, we get:

{    "Id": 10000,    "Name": "admin"}

This form is generally not the usual form of JSON, we are usually more accustomed to using JSON key initials lowercase, then the structure tag can be useful:

type Account struct {    Id string `json:"id"`    Name string `json:"name"`}var account = Account{    Id: 10000,    Name: "admin",}

When we convert to JSON at this time, we get the following result:

{    "id": 10000,    "name": "admin"}

Embed a key-value storage

For the sake of simplicity, we use a simple key-value storage boltdb, which is a go language embedded Key-value database. It provides a fast and reliable database for applications, so we don't need complex databases such as MySQL or Postgres.

We can get its source code through go get:

go get github.com/boltdb/bolt

Next, we create a dbclient directory below the Accountservice directory and create a boltclient.go file underneath it. For the convenience of subsequent simulations, we declare an interface that defines the contracts we need to fulfill:

package dbclientimport (    "github.com/callistaenterprise/goblog/accountservice/model")type IBoltClient interface() {    OpenBoltDb()    QueryAccount(accountId string) (model.Account, error)    Seed()}// 真实实现type BoltClient struct {    boltDb *bolt.DB}func (bc *BoltClient) OpenBoltDB() {    var err error    bc.boltDB, err = bolt.Open("account.db", 0600, nil)    if err != nil {        log.Fatal(err)    }}

The above code declares a iboltclient interface, which specifies that the contract for that interface has three methods. We declare a specific boltclient type, and for the time being it implements only the Openboltdb method. This method of implementing an interface can suddenly seem a little strange, binding a function on a struct. This is the feature of the Go Language interface implementation. The other two methods are skipped for the time being.

We now have the boltclient structure, and then we need to have an instance of the struct in a location somewhere in the project. Then we'll put it where we're going to use it and put it in our Goblog/accountservice/service/handlers.go file. We first create this file and then add an instance of Boltclient:

package serviceimport (    "github.com/callistaenterprise/goblog/accountservice/dbclient")var DBClient dbclient.IBoltClient

Then update the MAIN.GO code to open the DB when it starts.

func main() {    fmt.Printf("Starting %v\n", appName)    initializeBoltClient()                 // NEW    service.StartWebServer("6767")}// Creates instance and calls the OpenBoltDb and Seed funcsfunc initializeBoltClient() {    service.DBClient = &dbclient.BoltClient{}    service.DBClient.OpenBoltDb()    service.DBClient.Seed()}

This will open the database when our microservices start. But there is nothing to do here. Let's add some code that will allow us to boot some accounts when the service starts.

Populate some accounts at startup

Open the Boltclient.go code file and add a seed method for boltclient:

Start seeding Accountsfunc (BC *boltclient) Seed () {initializebucket () seedaccounts ()}//creates an "Accountbuck Et "in our boltdb. It would overwrite any existing bucket of the same name.func (BC *boltclient) initializebucket () {Bc.boltDB.Update (func (TX *bolt. TX) Error {_, err: = Tx. Createbucket ([]byte ("Accountbucket")) if err! = Nil {return FMT.  Errorf ("Create Bucket failed:%s", err)} return nil})}//Seed (n) Make-Believe account objects into the Acountbucket bucket.func (BC *boltclient) seedaccounts () {total: = + for I: = 0, i < total; i++ {//G Enerate a key 10000 or larger key: = StrConv. Itoa (10000 + i)//Create An instance for our account struct ACC: = model. account{Id:key, Name: "Person_" + StrConv. Itoa (i),}//Serialize the struct to JSON jsonbytes, _: = json. Marshal (ACC)//Write the data to the Accountbucket bc.boltDB.UpDate (func (TX *bolt). TX) Error {b: = tx. Bucket ([]byte ("Accountbucket")) Err: = B.put ([]byte (Key), jsonbytes) return err})} FM t.printf ("Seeded%v fake accounts...\n", total)}

Our seed method above first creates a bucket using the "Accountbucket" string, and then creates 100 initialized accounts in a row. The account ID is 10000~10100, respectively, and its name is person_i (i = 0 ~ 100).

We have already called the Seed () method in Main.go, so we can run the current program at this time to see how it works:

> go run *.goStarting accountserviceSeeded 100 fake accounts...2017/01/31 16:30:59 Starting HTTP service at 6767

Very good! Then we pause the execution and use CTRL + C to let the service stop first.

Add a Query method

Next we can add a query method to Boltclient.go to complete the DB API.

Func (BC *boltclient) queryaccount (AccountId string) (model. Account, error) {//Allocate a empty account instance we'll let json.    Unmarhal populate for us in a bit. Account: = model. account{}//Read an object from the bucket using Boltdb.view err: = Bc.boltDB.View (func (TX *bolt. TX) Error {//Read the bucket from the DB b: = Tx. Bucket ([]byte ("Accountbucket"))//Read The value identified by our accountId supplied as []byte accountbyte S: = B.get ([]byte (AccountId)) if accountbytes = = Nil {return FMT. Errorf ("No account found for" + accountId)}//Unmarshal the returned bytes to the account struct we cre Ated at//the top of the function JSON.    Unmarshal (Accountbytes, &account)//return nil to indicate nothing went wrong, e.g no error Return nil })//If There were an error, return the error If err! = Nil {return model. account{}, err}//Return the AccouNT struct and nil as error. Return account, nil}

This method is also relatively simple, based on the request parameter AccountId in the db we initialized earlier to find information about this account. If you successfully find the relevant account, return the JSON data for this account, otherwise it will return nil.

Provide account service via HTTP

Let's modify the/accounts/{accountid} route declared in the/service/routes.go file, and let it return one of the records of the account we populate. The code is modified as follows:

package serviceimport "net/http"// Defines a single route, e.g. a human readable name, HTTP method, pattern the function that will execute when the route is called.type Route struct {    Name        string    Method      string    Pattern     string    HandlerFunc http.HandlerFunc}// Defines the type Routes which is just an array (slice) of Route structs.type Routes []Routevar routes = Routes{    Route{        "GetAccount",             // Name        "GET",                    // HTTP method        "/accounts/{accountId}",  // Route pattern        GetAccount,    },}

Next, we update the next/service/handlers.go and create a getaccount function to implement the HTTP processor function signature:

var DBClient dbclient.IBoltClientfunc GetAccount(w http.ResponseWriter, r *http.Request) {    // Read the 'accountId' path parameter from the mux map    var accountId = mux.Vars(r)["accountId"]        // Read the account struct BoltDB    account, err := DBClient.QueryAccount(accountId)        // If err, return a 404    if err != nil {        w.WriteHeader(http.StatusNotFound)        return    }        // If found, marshal into JSON, write headers and content    data, _ := json.Marshal(account)    w.Header().Set("Content-Type", "application/json")    w.Header().Set("Content-Length", strconv.Itoa(len(data)))    w.WriteHeader(http.StatusOK)    w.Write(data)}

The code above is to implement the processor function signature, and when Gorilla detects that we are requesting/accounts/{accountid}, it will route the request to this function. Let's run our service below.

> go run *.goStarting accountserviceSeeded 100 fake accounts...2017/01/31 16:30:59 Starting HTTP service at 6767

Then another window is opened, and curl requests a request for AccountId of 10000:

> curl http://localhost:6767/accounts/10000{"id":"10000","name":"Person_0"}

Great, our microservices are now able to provide some simple data on the fly. You can try to use any number from AccountId to 10000 to 10100, and get a different JSON.

Occupy Space and performance

(Footprint is explained here as taking up space, memory space).

In the second part, we see that the space occupancy information is as follows in the galtling pressure measurement situation:

Again we do a pressure test on the service, the space occupied by the following:

We can see that after the increase of boltdb, memory consumption from 2.1MB to 31.2MB, increased by about 30MB, not too bad.

1000 requests per second, the approximate usage per CPU core is 10%,BOLTDB and JSON serialization overhead is not very obvious, very good! By the way, our previous Java process was 3 times times more CPU-galting under the pressure test.

The average response time is still less than 1 milliseconds. Maybe we need to use heavier pressure tests to test, we try to use 4K per second request? (Note that we may need to increase the number of file processing available at the OS level).

The memory becomes more than 118MB, which is basically 4 times times higher than the original. The increase in memory is almost due to the go language runtime or because gorilla increases the number of internal goroutine used for service requests, thus increasing the load.

The CPU basically stays at 30%. I run on a 16GB Ram/core i7 notebook, and I think I/O or file handles are a performance bottleneck faster than CPUs.

The average throughput last up to 95% requests between 1ms~3ms. It is true that throughput has been affected by the 4K/S request, but it is quite good that the small Accountservice service uses BOLTDB.

Last Words

In the next section, we will explore the use of Goconvey and simulated BOLTDB clients for unit testing.

Reference links

• English address
• Go fabric and interface
• Micro-Service Series entrance
• Next section