Go microservices-part sixth-health check

This is a creation in Article, where the information may have evolved or changed.

Part VI: Go microservices-Health Check

As our microservices become more and more complex, it is important to have a mechanism that lets Docker swarm know that a service is healthy. Therefore, this article focuses on how to add a health check for a microservices service.

If the Accountservice micro-service does not have the following capabilities, it is useless:

• provides HTTP services.
• Connect to its own database.

Health Terminal monitoring mode

Implement feature checking in your application, and external tools can be implemented with regular access through exposed endpoints. This helps verify that the application and the service are performing correctly.

The usual way to handle these situations in microservices is to provide health check routing (a good article for Azure Docs), in our case, since we are HTTP-based, we should map a/health, and respond 200 if it will, You may need to bring some machine possibilities to explain what is OK. If there is a problem, return a non-200 response, as well as possible unhealthy causes. Note that some people should return 200 for the check failure and the information to attach the error in the response payload. I agree with that, but in order to simplify, we use non-200来 checks directly in this article. So we need to add a/health route in the Accountservice.

Source

As always, we can check out the corresponding branch from git to get some of the changed code.

Https://github.com/callistaen ...

Add a check to Boltdb

Our service is useless if we cannot access the underlying database properly. Therefore, we need to add an interface check () to Iboltclient.

type IBoltClient interface {    OpenBoltDb()    QueryAccount(accountId string) (model.Account, error)    Seed()    Check() bool              

The check method may seem simple, but it still works in this article. It returns TRUE or false depending on whether the boltdb is available.

Our check implementation in the Boltclient.go file is not very realistic, but it is enough to explain the concept.

// Naive healthcheck, just makes sure the DB connection has been initialized.func (bc *BoltClient) Check() bool {    return bc.boltDB != nil}

The simulation implementation in Mockclient.go also follows our extension/test (stretchr/testify) Standard mode:

func (m *MockBoltClient) Check() bool {    args := m.Mock.Called()    return args.Get(0).(bool)}

Add/health Route

It's very straightforward here. We add the following route directly inside the Service/routes.go:

var routes = Routes{    Route{        "GetAccount",             // Name        "GET",                    // HTTP method        "/accounts/{accountId}",  // Route pattern        GetAccount,    },    Route{        "HealthCheck",        "GET",        "/health",        HealthCheck,    },}

/health request for Healthcheck to handle. Here's what's Healthcheck:

func HealthCheck(w http.ResponseWriter, r *http.Request) {    // Since we're here, we already know that HTTP service is up. Let's just check the state of the boltdb connection    dbUp := DBClient.Check()    if dbUp {        data, _ := json.Marshal(healthCheckResponse{Status: "UP"})        writeJsonResponse(w, http.StatusOK, data)    } else {        data, _ := json.Marshal(healthCheckResponse{Status: "Database unaccessible"})        writeJsonResponse(w, http.StatusServiceUnavailable, data)    }}func writeJsonResponse(w http.ResponseWriter, status int, data []byte) {    w.Header().Set("Content-Type", "application/json")    w.Header().Set("Content-Length", strconv.Itoa(len(data)))    w.WriteHeader(status)    w.Write(data)}type healthCheckResponse struct {    Status string `json:"status"`}

The Healthcheck function agent checks the DB state, which is the check () method added in Dbclient. If OK, we create a healthcheckresponse struct. Note that the first letter is lowercase and is visible only within the scope of the module. We also implemented a method of writing HTTP responses so that the code would look concise.

Run

Run the modified code:

> go run *.goStarting accountserviceSeeded 100 fake accounts...2017/03/03 21:00:31 Starting HTTP service at 6767

Then open a new window and use Curl to access the/health interface:

> curl localhost:6767/health{"status":"UP"}

It works!

Docker Health Check-up

Next, we'll use Docker's healthcheck mechanism to let Docker swarm check our service activity. This is done by adding a line to the Dockerfile file:

FROM iron/baseEXPOSE 6767ADD accountservice-linux-amd64 /ADD healthchecker-linux-amd64 /HEALTHCHECK --interval=1s --timeout=3s CMD ["./healthchecker-linux-amd64", "-port=6767"] || exit 1ENTRYPOINT ["./accountservice-linux-amd64"]

What is HEALTHCHECKER-LINUX-AMD64? We need to help docker a little bit because Docker does not provide us with HTTP clients or something like that to perform health checks. Instead, the Healthcheck directive in dockerfile specifies a command (CMD) that should perform a call to the/health route. Depending on the exit code of the running program, Docker determines whether the service is healthy. If too many subsequent health checks fail, Docker swarm will kill the container and start a new container.

The most common way to implement a real health check looks like curl. However, this requires our base Docker image to actually install curl (or any underlying dependency), and at this point we don't really want to handle it. Instead, let the go language brew our own health Check applet.

Create a Health Check program

It's time to create a new sub-project under Src/github.com/callistaenterprise/goblog.

mkdir healthchecker

Then create a main.go file under this directory with the following contents:

package mainimport (    "flag"    "net/http"    "os")func main() {    port := flag.String("port", "80", "port on localhost to check")    flag.Parse()    resp, err := http.Get("http://127.0.0.1:" + *port + "/health") // 注意使用 * 间接引用    // If there is an error or non-200 status, exit with 1 signaling unsuccessful check.    if err != nil || resp.StatusCode != 200 {        os.Exit(1)    }    os.Exit(0)}

The amount of code is not very large, what does it do?

• Use the flag package to read the-port command-line arguments. If not specified, fallback uses the default value of 80.
• Executes an HTTP GET request http://127.0.0.1:[port]/health.
• If an HTTP request error occurs, the status code is not 200 and exits with the eject code 1. Otherwise exit with exit code 0. 0 = = Success, 1 = = fail.

Let's try, if we've stopped Accountservice, run it again and run Healthchecker.

go build./accountservice

Then run this program:

> cd $GOPATH/src/github.com/callistaenterprise/goblog/healtchecker> go run *.goexit status 1

We forgot to specify the port number above, so it uses the default 80 port. Let's do it again:

> go run *.go -port=6767>

There is no output here to indicate that our request was successful. Well, then we build a linux/amd64 binary and add it to accountservice by adding Healthchecker binary to the Dockerfile file. We continue to rebuild and deploy automatically using copyall.sh scripts.

#!/bin/bashexport GOOS=linuxexport CGO_ENABLED=0cd accountservice;go get;go build -o accountservice-linux-amd64;echo built `pwd`;cd ..// NEW, builds the healthchecker binarycd healthchecker;go get;go build -o healthchecker-linux-amd64;echo built `pwd`;cd ..export GOOS=darwin   // NEW, copies the healthchecker binary into the accountservice/ foldercp healthchecker/healthchecker-linux-amd64 accountservice/docker build -t someprefix/accountservice accountservice/

Finally, we need to do one more thing, is to update the Accountservice dockerfile. Its full content is as follows:

FROM iron/baseEXPOSE 6767ADD accountservice-linux-amd64 /# NEW!! ADD healthchecker-linux-amd64 /HEALTHCHECK --interval=3s --timeout=3s CMD ["./healthchecker-linux-amd64", "-port=6767"] || exit 1ENTRYPOINT ["./accountservice-linux-amd64"]

We have attached the following content:

• Adding an Add directive ensures that the Healthchecker binary is included in the image.
• The Healthcheck statement specifies our binaries and parameters, telling Docker to perform a health check every 3 seconds and accept a 3-second timeout.

Deploy Health Check Service

Now we are ready to deploy our updated Accountservice service with Healthchecker. If you want to be more automatic, add these two lines to the copyall.sh file, and each time it runs, it automatically deletes the accountservice from the Docker swarm and re-creates it.

docker service rm accountservicedocker service create --name=accountservice --replicas=1 --network=my_network -p=6767:6767 someprefix/accountservice

So now run./copyall.sh, wait a few seconds, all build updated well. Then we use Docker PS to check the status of the container and we can enumerate all the running containers.

> docker psCONTAINER ID        IMAGE                             COMMAND                 CREATED        STATUS1d9ec8122961        someprefix/accountservice:latest  "./accountservice-lin"  8 seconds ago  Up 6 seconds (healthy)107dc2f5e3fc        manomarks/visualizer              "npm start"             7 days ago     Up 7 days

We look for the "(healthy)" text below the status header. The service does not have a health instruction configured Healthcheck completely.

Deliberately causing a failure

To make things a little more interesting, we add a testable API that allows us to let endpoints play unhealthy purposes. In the Routes.go file, declare another route.

var routes = Routes{    Route{        "GetAccount", // Name        "GET",        // HTTP method        "/accounts/{accountId}", // Route pattern        /*func(w http.ResponseWriter, r *http.Request) {            w.Header().Set("Content-Type", "application/json; charset=UTF-8")            w.Write([]byte("{\"result\":\"OK\"}"))        },*/        GetAccount,    },    Route{        "HealthCheck",        "GET",        "/health",        HealthCheck,    },    Route{        "Testability",        "GET",        "/testability/healthy/{state}",        SetHealthyState,    },}

This route (do not have such a route in the production service!) Provides a rest-ish route for the purpose of troubleshooting health checks. The Sethealthystate function is in the Goblog/accountservice/handlers.go file with the following code:

var isHealthy = true // NEWfunc SetHealthyState(w http.ResponseWriter, r *http.Request) {    // Read the 'state' path parameter from the mux map and convert to a bool    var state, err = strconv.ParseBool(mux.Vars(r)["state"])        // If we couldn't parse the state param, return a HTTP 400    if err != nil {        fmt.Println("Invalid request to SetHealthyState, allowed values are true or false")        w.WriteHeader(http.StatusBadRequest)        return    }        // Otherwise, mutate the package scoped "isHealthy" variable.    isHealthy = state    w.WriteHeader(http.StatusOK)}

Finally, Ishealthy Boolean is used as the check condition for the Healthcheck function:

func HealthCheck(w http.ResponseWriter, r *http.Request) {        // Since we're here, we already know that HTTP service is up. Let's just check the state of the boltdb connection        dbUp := DBClient.Check()                if dbUp && isHealthy {              // NEW condition here!                data, _ := json.Marshal(                ...        ...        }

Restart Accountservice.

> cd $GOPATH/src/github.com/callistaenterprise/goblog/accountservice> go run *.goStarting accountserviceSeeded 100 fake accounts...2017/03/03 21:19:24 Starting HTTP service at 6767

A new Healthcheck call is then generated in the new window.

> cd $GOPATH/src/github.com/callistaenterprise/goblog/healthchecker> go run *.go -port=6767

The first attempt succeeds, and then we change the state of Accountservice by using the following Curl request testability.

> curl localhost:6767/testability/healthy/false> go run *.go -port=6767exit status 1

It's working! Then we run it in the Docker swarm. Use copyall.sh to rebuild and redeploy accountservice.

> cd $GOPATH/src/github.com/callistaenterprise/goblog> ./copyall.sh

As always, wait for Docker swarm to redeploy "Accountservice", using the newly built "Accountservice" container image. Then, run Docker PS to see if the service is up and running with health.

> docker psCONTAINER ID    IMAGE                            COMMAND                CREATED         STATUS 8640f41f9939    someprefix/accountservice:latest "./accountservice-lin" 19 seconds ago  Up 18 seconds (healthy)

Note the container ID and created fields. You can increase the testability API on your Docker swarm. (My IP is: 192.168.99.100).

> curl $ManagerIP:6767/testability/healthy/false>

Then we run the Docker PS command again in a few seconds.

> docker psCONTAINER ID        IMAGE                            COMMAND                CREATED         STATUS                                                             NAMES0a6dc695fc2d        someprefix/accountservice:latest "./accountservice-lin" 3 seconds ago  Up 2 seconds (healthy)

As you can see, here's a new container ID and created and status. What really happens is that Docker swarm monitors three consecutive failed health checks (default values for retries) and immediately determines that the service is unhealthy and needs to be replaced with a new instance, which happens entirely without any management intervention.

Summarize

In this section, we use a simple/health Routing and Healthchecker program combined with Docker's healthcheck mechanism to demonstrate how this mechanism allows Docker swarm to automatically handle unhealthy services for us.

In the next chapter, we delve into the mechanism of Docker swarm, where we focus on two key areas-service discovery and load balancing.

Reference links

• 6th part of English
• Health Endpoint Check
• Topic Home
• Next section