Go build microservices framework based on GRPC-integrated opentracing

This is a creation in Article, where the information may have evolved or changed.

1. Overview

There is a scenario where a request will be processed by multiple services after the microservices split, and it will be difficult to troubleshoot if a service fails on the requested link.
We may need to take the requested service, look at the log for analysis, and when the service has dozens of hundred instances, this is certainly scary. Therefore, in order to solve this problem, call chain tracking came into being.

2.opentracing

1.1 Opentracing Effect

Call chain tracking was first proposed by Googel in dapper this paper, opentracing mainly defines the relevant protocols and interfaces, so that each language as long as the Opentracing interface and protocol to achieve data escalation, then the call information can be unified collection.

As shown, the interface might first go through the web framework and then invoke the Auth service, numbering the requested service through the call chain, gathering it together to form a logical link so that we can see what services are being requested and thus form the topology of the service dependency.

As above, the total chain routing of each segment of the link, each link represents the service, time-consuming can be used to analyze the system bottleneck, when a request is slow to return, you can troubleshoot a section of the link time-consuming situation, so as to analyze which service has a high latency, this time to specific services to analyze specific problems.

1.2 opentraing Key Terminology

• Traces (call chain)

The link of a call, by the Traceid unique flag, such as a single request, is typically a trace,trace consisting of spans of all paths.

• Spans (call span)

No one service will span, same as each span by Spanid unique flag.

• span tags (span tags)

span labels, such as a span that calls Redis, can set up a Redis label so that by searching for the Redis keyword, we can query all relevant spans and trace.

• Baggage Item (with data)

Additional data, made up of key:value, can give more descriptive information to the call chain through additional data, but the additional data should be as few as possible given the transmission problem.

1.3 Jaeger & Zipkin

The current implementation of open source has Zipkin and Jaeger

• Zipkin

Zipkin is mainly written by Java, through the various language of the escalation library implementation of the data on the report Collector,collector and then store the data, and through the API to provide to the previous UI display.

• Jaeger

Jaeger is implemented by go, developed by Uber, is currently a cloud native project, and the process is similar to Zipkin, with the addition of jager-agent, a component that is officially recommended to deploy one per machine, Through this component and then the data on the report collector storage display, in addition, the inside did a zipkin adaptation, in fact, they used to be zipkin, for the hair behind to build their own wheels? See their explanations. Link

In general, both can basically meet the opentracing function, the specific choice can be combined with their own technology stack and hobby.

2. GRPC Integrated opentracing

GRPC Integrated opentracing is not difficult because the GRPC server and the caller declare unaryclientinterceptor and Unaryserverinterceptor two callback functions respectively, so only the two callback functions need to be rewritten. and invoke the Opentracing interface in the overridden callback function to escalate.
When initialized, pass in the rewritten callback function, and at the same time two select Initialize Jager or Zipkin, then you can turn on the distributed call chain tracking trip.

Complete code See Grpc-wrapper.

2.1 Client Side

Opentracingclientinterceptor rewrite client ' s interceptor with open Tracingfunc Opentracingclientinterceptor (tracer Opentracing. Tracer) Grpc. Unaryclientinterceptor {return func (CTX context). Context, method String, req, resp interface{}, CC *GRPC. Clientconn, Invoker Grpc. Unaryinvoker, opts ... grpc.        Calloption,) error {//Get Spancontext from the context, if the upper level does not turn on tracking, then create a new//trace here, if the upper layer already has, Test creates a child span. var parentctx opentracing. Spancontext if Parent: = opentracing. Spanfromcontext (CTX); Parent! = Nil {parentctx = parent. Context ()} Clispan: = Tracer. Startspan (method, opentracing. Childof (PARENTCTX), wrapper. Tracingcomponenttag, ext.  Spankindrpcclient,) defer clispan.finish ()//Take out the metadata data that was previously placed in the context, and if not, create a new metadata MD, OK: = metadata. Fromoutgoingcontext (CTX) if!ok {md = metadata. New (Nil)} else {            MD = Md. Copy ()} Mdwriter: = MDREADERWRITER{MD}//injects trace data into metadata err: = Tracer. Inject (Clispan.context (), opentracing. Textmap, Mdwriter) if err! = Nil {grpclog. Errorf ("Inject to metadata err%v", err)}//Load metadata data into the context CTX = metadata.        Newoutgoingcontext (CTX, MD)//Use a context with tracking data to make a GRPC call.        Err = Invoker (CTx, method, req, RESP, CC, opts ...) If err! = Nil {clispan.logfields (log. String ("Err", err. Error ())} Return Err}}

2.2 Server Side

Opentracingserverinterceptor rewrite server ' s interceptor with open Tracingfunc Opentracingserverinterceptor (tracer Opentracing. Tracer) Grpc. Unaryserverinterceptor {return func (CTX context). Context, req interface{}, Info *grpc. Unaryserverinfo, Handler Grpc. Unaryhandler,) (Resp interface{}, err Error) {//Remove metadata MD from context, OK: = metadata. Fromincomingcontext (CTX) if!ok {md = metadata. New (NIL)}//Extract the final data from metadata and create a span object Spancontext, err: = Tracer. Extract (opentracing. Textmap, MDREADERWRITER{MD}) if err! = Nil && Err! = opentracing. Errspancontextnotfound {Grpclog. Errorf ("Extract from metadata err%v", err)}//Initialize the server-side span Serverspan: = tracer. Startspan (info. Fullmethod, ext. Rpcserveroption (Spancontext), wrapper. Tracingcomponenttag, ext.        Spankindrpcserver,)Defer serverspan.finish () CTX = opentracing. Contextwithspan (CTX, Serverspan)//Call return handler (CTX, req)} In the context incoming application code with tracing

Since opentracing defines the relevant interfaces, Jaeger and Zipkin are implemented accordingly, so Jaeger can also be used to escalate the zipkin.

3. Effects

Jaeger Service Homepage Information

Each call chain information

4. Reference

Zipkin
Jaeger
Opentracing
Grpc-wrapper