On-the-Go Micro-service Setup (vi)-Health Check

This is a creation in Article, where the information may have evolved or changed.

Section Sixth: Health Check
Original address
Reprint please specify the original and translation address

As our microservices become more complex, it is important for Docker swarm to know that our services are running well. Let's look at how to view service health.
For example, our Accountservice service will not work if HTTP or linked databases cannot be served.
The best way to do this is to provide a healthcheck access point. We are based on HTTP, so map to/health, if it works well, return HTTP 200 colleague some explain what is good information. If there is a problem, non-HTTP 200 returns, and explain where it's not good. Some people think that all should return 200, and then return an error message. I agree, but in this simple example, I will return with a non 200.

Code

Like, you can branch directly to this part.

git checkout P6

Join the Boltdb Check

Our services if we cannot connect to database it will be useless, so we join the function Check ()

type IBoltClient interface {b    OpenBoltDb()    QueryAccount(accountId string) (model.Account, error)    Seed()    Check() bool //new}

This function may be simple, but enough, he will return True/false based on whether Boltdb can connect.

func (bc *BoltClient) Check() bool {    return bc.boltDB != nil}

Mocked code in MACKCLIENT.GO compliance with stretchr/testify form

func (m *MockBoltClient) Check() bool {    args := m.Mock.Called()    return args.Get(0).(bool)

Join/health Path

Very direct, we joined in the Routes.go.

Route{    "HealthCheck",    "GET",    "/health",    HealthCheck},

We use the function Healthcheck to process the request, and we add the function to the Handler.go:

  func healthcheck (w http. Responsewriter, R *http. Request) {//Since we ' re here, we already know the HTTP service is up. Let's just check the state of the BOLTDB connection dbup: = Dbclient.check () If Dbup {data, _: = json. Marshal (healthcheckresponse{status: "Up"}) writejsonresponse (W, http. Statusok, data)} else {data, _: = json. Marshal (healthcheckresponse{status: "Database unaccessible"}) Writejsonresponse (W, http. statusserviceunavailable, data)}}func Writejsonresponse (w http. Responsewriter, status int, data []byte) {W.header (). Set ("Content-type", "Application/json") W.header (). Set ("Content-length", StrConv. Itoa (len (data))) W.writeheader (status) w.write (data)}type healthcheckresponse struct {status string ' JSON: ' Status ' '}  

The Healthcheck function checks the database condition with the check () function. If normal, we return an instance of the Healthcheckresponse structure. Note the lowercase initials, This structure can only be used in this package. We also extract the code that returns the result into a function that lets us not repeat the code.

Run

In the Blog/accountservice folder, run:

> go run *.goStarting accountserviceSeeded 100 fake accounts...2017/03/03 21:00:31 Starting HTTP service at 6767

Curl this/health path

> curl localhost:6767/health{"status":"UP"}

Docker Healthcheck

Next, we use Docker's health check mechanism to check our services. Add the following command at Dockerfile:

HEALTHCHECK --interval=5s --timeout=5s CMD["./healthchecker-linux-amd64", "-port=6767"] || exit 1

What is HEALTHCHECKER-LINUX-AMD64? Docker does not know how to do this health check, we need to help, we are in the cmd command input to guide to the/health path. According to exit code, Docker will judge whether the service is good or not. If too many checks fail, swarm turns off the container and opens a new instance
The most common health checks use curl, however this requires our Docker image to install curl. Here we will use go to execute this applet.

Create a Helathchecker program

Add a folder under Goblog

mkdir healthchecker

Join Main.go

package mainimport (    "flag"    "net/http"    "os")func main() {    port := flag.String("port", "80", "port on localhost to check")     flag.Parse()    resp, err := http.Get("http://127.0.0.1:" + *port + "/health")    // Note pointer dereference using *        // If there is an error or non-200 status, exit with 1 signaling unsuccessful check.    if err != nil || resp.StatusCode != 200 {        os.Exit(1)    }    os.Exit(0)}

Code is not much, mainly do:

• Read the-port=nnnn command parameter with the built-in flags, if not, use the default port of 80
• Start HTTP GET request 127.0.0.1:[port]/health
• If there is an error or the return status is not 200, exit the same non-0 value, 0== succeeds, >0== fails

Try, if you stop Accountservice, start with Go run *.go, or compile it go build./accountservice
Then go back to the background to run Healthchecker

> cd $GOPATH/src/github.com/callistaenterprise/goblog/healthchecker> go run *.goexit status 1

Oops! We forgot to give the port number. Try again.

> go run *.go -port=6767>

No output indicates that we succeeded. OK, let's compile a linux/amd64 binary and add it to accountservice by adding Healthchecker in Dockerfile. We use the copyall.sh script to do:

#!/bin/bashexport GOOS=linuxexport CGO_ENABLED=0cd accountservice;go get;go build -o accountservice-linux-amd64;echo built `pwd`;cd ..// NEW, builds the healthchecker binarycd healthchecker;go get;go build -o healthchecker-linux-amd64;echo built `pwd`;cd ..export GOOS=darwin   // NEW, copies the healthchecker binary into the accountservice/ foldercp healthchecker/healthchecker-linux-amd64 accountservice/docker build -t someprefix/accountservice accountservice/

At the same time, we update Accountservice's dockerfile:

FROM iron/baseEXPOSE 6767ADD accountservice-linux-amd64 /# NEW!! ADD healthchecker-linux-amd64 /HEALTHCHECK --interval=3s --timeout=3s CMD ["./healthchecker-linux-amd64", "-port=6767"] || exit 1ENTRYPOINT ["./accountservice-linux-amd64"]

Part of the Join

• Add an Add statement to determine that Healthchecker is added to the mirror.
• The Healthcheck statement tells Docker to execute once every 3s with a timeout of 3s

Deploying Healthcheck

Now we can deploy the Accountservice with the healthchecking. Automation to do these things, add two lines to the copyall.sh:

docker service rm accountservicedocker service create --name=accountservice --replica=1 --network=my_network -p=6767:6767someprefix/accountservice

Run./copyall.sh Wait a few seconds, then check the container status, Docker PS:

> docker psCONTAINER ID        IMAGE                             COMMAND                 CREATED        STATUS                1d9ec8122961        someprefix/accountservice:latest  "./accountservice-lin"  8 seconds ago  Up 6 seconds (healthy)107dc2f5e3fc        manomarks/visualizer              "npm start"             7 days ago     Up 7 days

We see the (Healthy) field in the status bar and there is no health check in the service that does not have this hint.

Look at the failure situation.

Let's add an API that can be tested to make the path unhealthy. In Routes.go, add a new path:

Route{        "Testability",        "GET",        "/testability/healthy/{state}",        SetHealthyState,},    

This path (you should not include him in the production environment) provides us with a way to let the health check fail. The Sethealthystate function is in Handlers.go:

var isHealthy = true // NEWfunc SetHealthyState(w http.ResponseWriter, r *http.Request) {        // Read the 'state' path parameter from the mux map and convert to a bool        var state, err = strconv.ParseBool(mux.Vars(r)["state"])                // If we couldn't parse the state param, return a HTTP 400        if err != nil {                fmt.Println("Invalid request to SetHealthyState, allowed values are true or false")                w.WriteHeader(http.StatusBadRequest)                return        }                // Otherwise, mutate the package scoped "isHealthy" variable.        isHealthy = state        w.WriteHeader(http.StatusOK)}

Restart Accountservice

func HealthCheck(w http.ResponseWriter, r *http.Request) {        // Since we're here, we already know that HTTP service is up. Let's just check the state of the boltdb connection        dbUp := DBClient.Check()                if dbUp && isHealthy {              // NEW condition here!                data, _ := json.Marshal(                ...        ...        }

Re-Request Healthcheck

> cd $GOPATH/src/github.com/callistaenterprise/goblog/accountservice> go run *.goStarting accountserviceSeeded 100 fake accounts...2017/03/03 21:19:24 Starting HTTP service at 6767

The first attempt succeeds, now change accountservice with Curl request to test path

> curl localhost:6767/testability/healthy/false> go run *.go -port=6767exit status 1

Works fine, let's run in Docker Swarm, recompile and deploy with copyall.sh

> cd $GOPATH/src/github.com/callistaenterprise/goblog> ./copyall.sh

Wait a minute, then run Docker PS to see our health services

> docker psCONTAINER ID    IMAGE                            COMMAND                CREATED         STATUS 8640f41f9939    someprefix/accountservice:latest "./accountservice-lin" 19 seconds ago  Up 18 seconds (healthy)

Note the container ID and created. Request the Test API, mine is 192.168.99.100

> curl $ManagerIP:6767/testability/healthy/false>

Now, run the Docker PS

> docker psCONTAINER ID        IMAGE                            COMMAND                CREATED         STATUS                                                             NAMES0a6dc695fc2d        someprefix/accountservice:latest "./accountservice-lin" 3 seconds ago  Up 2 seconds (healthy)

Look, a new container ID and a new created and status timestamp. Because the swarm is checked every three seconds and the service is not healthy, it is replaced with a new service and does not require an administrator's intervention.

Summarize

We add a simple/health path and some Docker health check mechanisms. Show how swarm is controlling non-health services.
In the next section, we'll dive into swarm, and we'll focus on microservices two architectures: Service discovery and load balancing.