How to do real-time monitoring? --Refer to Spring Boot implementation

Transferred from: http://blog.csdn.net/xiaoyu411502/article/details/48129057

With the popularity of microservices, we are now more inclined to cut large applications into multiple microservices and service-to-RPC calls than a large one before. There are many benefits to the microservices architecture, such as stable service changes that are not affected by non-stable services, more convenient for different people to manage, and more targeted for publishing, scaling, and more. However, this is not without cost, the additional cost of the most likely is the operation and maintenance costs.

We maintain a product, consisting of 7 micro-services, their respective responsibilities, assume the uplink, downlink, synchronization and other duties, I like this architecture, but also face a little trouble. Each time we publish one or more of the services, we need to verify the health of the service, limit the situation, 7 x (Domestic environment + foreign environment) x (pre-release environment + production environment), a total of 28 times to verify! I would like to have a simple, standard, automated way to verify that these services are healthy. Of course, verifying health is not to run a complete regression test, that is in the test environment needs to be done, the health check is basically just concerned about the environment is OK, the core of the one or two use cases is OK. Since the code deployed to pre-release or online code is consistent with offline testing, there is no need to repeat the verification of the various functions, and the focus should be on the environment, which is clearly different from the line and the line. As for the environmental difference, it is usually disk, database, other distributed services and so on.

In addition, I also hope that the health check interface of all services is exactly the same, no one wants to check service A when using Url/ok, check service B when using Url/good.

I have tried to define a health check protocol so that all services expose an HTTP interface Http://172.20.10.2/health.json, and the content returned contains the basic state of this service.

Looking at spring boot these days, I found that it was well integrated with the features I wanted, and it looked simpler, so I just threw away the protocols I had defined, and using spring boot, Spring boot had a concept called endpoint, with each end Point is a very simple HTTP interface that allows users to monitor and even interact with the Spring Boot application via endpoint. The simplest of these endpoint is health, which, by adding the necessary spring boot dependencies, allows the user to view the basic state of the spring boot app with the.

http://localhost:8080/health{    "status":"Up"}   

Here we see the status of the service is up, but perhaps this check is too simple, for example, my service relies on other external services, one tair, one is TFS, both are strong dependencies, if they have problems, my service should be down state, in Spring Boot, This can be extended:

@ComponentPublicclass myhealth implements  Healthindicator { @Override public health health () {  return new health.builder ()   Withdetail ( "Tair",  " Timeout ") //some logic check tair  Withdetail ( "TFS",  "OK") //some logic check tfs Span class= "indent" > status (   Build ();}           

A more comprehensive check can be achieved with the addition of a bean implementation healthindicator, and now access to the health endpoint is as follows:

http://localhost:8080/health{    "status""Tair" "TFS" "OK"}    

As long as there are some basic environmental checks in each service, I can quickly complete 7 x (Domestic environment + foreign environment) x (pre-release environment + production environment) health check with a few lines of script, and it is very convenient to locate the environment problem if there are any service problems.

This monitoring is real-time, which is very important. In the actual work we actually have a very perfect system monitoring platform, the platform can provide CPU, memory, disk, network IO, JVM and so on a variety of very comprehensive information, the advantages of this platform has historical trend record, there is a summary, there is a comparison, disadvantage is not enough real-time, usually only see the data 5 minutes ago. Therefore, it is not enough to wait for such a system to monitor the platform when releasing the service and expanding.

In addition to the health endpoint, Spring Boot also provides more than 10 other endpoint, which are designed for operations, such as shutdown endpoint to turn off services, beans endpoint to view all Spring Bean, I would like to elaborate on metrics this endpoint.

Default Access Metrics We get a lot of information, including the number of threads in the JVM, memory, GC data, and so on ... these are system-level data, but in fact we can collect real-time business data through metrics, such as the number of user logins per minute, the number of file synchronizations per minute, Real-time cache hit ratio ... Wait a minute.

The implementation is like this:

@ComponentPublicClassMymetric { PrivateFinal Counterservice Counterservice; PrivateFinal Gaugeservice Gaugeservice; @Autowired Public Mymetric (Counterservice counterservice, Gaugeservice gaugeservice) {  this.counterservice = counterservice;   this.gaugeservice = gaugeservice; } public void examplecountermethod () {  this.counterservice.increment ( " Login.count ");   //reset each Minute} public void examplegaugemethod () {   this.gaugeservice.submit ( " Cache.hit ", 80.0); }              /span>           

Spring Boot has built-in two service,counterservice that can be used to do simple cumulative subtraction, Gaugeservice can be used to store simple double values, and the data is stored in memory.

Now the effect of accessing metrics endpoint is this:

$ curlhttp://localhost:8080/metrics{  "Counter.login.count":42,  "Counter.status.200.beans" : 1,   "Counter.status.200.metrics" :  9,  "Counter.status.200.root" : 4,  80.0, : Span class= "number" >55, : Span class= "number" >12, : 4, ...}            

Spring Boot metrics endpoint with a lot of information, and here we focus only on custom data.

If the core business data of all the services is exposed through metrics, all we have to do is to access the data through some data visualization JavaScript components and make a Dashboard, so that we can see the real-time state of the system through such a Dashboard.

Spring Boot endpoints with a strong DevOps color, "You build it, you run it", development not only to care about how to implement the function, but also need to care about the status of service online running, if the lack of real-time monitoring, maintenance online service must be a nightmare. If you are based on the spring boot development service, that only needs a little expansion, real-time monitoring is sufficient, and even if you do not use spring boot, it is not complicated to implement similar ideas yourself.

Reference link http://docs.spring.io/spring-boot/docs/1.1.x/reference/htmlsingle/#production-ready-endpoints

How to do real-time monitoring? --Refer to Spring Boot implementation (GO)