Mutual transformation of Protobuffer, GRPC and restful grpc

This is a creation in Article, where the information may have evolved or changed.

Document

• GRPC Chinese Documents
• Grpc-gateway,restful and Grpc Conversion libraries
• PROTOBUF official website

Protobuf

Google Protocol Buffer (abbreviated as PROTOBUF) is a lightweight and efficient structured data storage format, platform-Independent, language-independent, extensible, and can be used in communication protocols and data storage areas.

Advantages

• Platform-Independent, language-independent, extensible;
• Provides a friendly dynamic library, easy to use;
• The parsing speed is about 20-100 times faster than the corresponding XML.
• The serialized data is very concise and compact, and the amount of data after serialization is approximately 1/3 to 1/10 compared to XML.

Installation

Refer to Golang using Protobuf

https://github.com/google/protobuf/releases // 下载并编译、安装go get github.com/golang/protobuf/proto   // golang的protobuf库文件// 插件go get github.com/golang/protobuf/protoc-gen-go  // 用于根据protobuf生成golang代码，语法 protoc --go_out=. *.proto

Grammar

Book/book.proto

syntax="proto3";package book;// import "xxx/xx.proto"// 出版社message Publisher{    required string name = 1}  // 书籍信息message Book {     required string name = 1;    message Author {        required string name = 1;        required string address = 1;    }    required Author author = 2;    enum BookType{        SCIENCE = 1 ;        LITERATURE = 2;    }    optional BookType type = 3;    optional Publisher publisher = 4}

• Syntax= "Proto3": Specifies the version of the PROTOBUF
• Package Book: Declare a registration, usually the same as the file directory name
• Import "Xxx/xx.proto": Importing other packages so that you can use the data structures of other packages
• Required, optional, repeated: Indicates whether the field must be populated, required indicates that it must be specified and only one can be specified, and when optional is optional, it can be specified or unspecified, but no more than one value is not specified, a null value is used. A field such as string is represented by a string, and repeated means that it can be repeated, similar to a list in a programming language
• Message Author: Defines a message structure body in a message body
• Enum: is an enumeration type struct body
• Number: identifier of the field, non-repeatable
• Data types: Int32, Int64, UInt32, UInt64, Sint32, Sint64, double, float, string, bool, bytes, enum, message, and more

Used in Golang

Protobuf using the Book.proto file above

and use the following command to generate the go file

protoc --go_out=. *.proto

Use in code

package mainimport (    b "book"    "github.com/golang/protobuf/proto")func main(){    ...    // 将实例转为proto编码    var b = &b.Book{Name:"xxx", Author:b.Author{Name:"yyy"}}    protoBook, err := proto.Marshal(b)    ...    // 讲proto编码转化为实例    var b2 b.Book    err = proto.Unmarshal(protoBook, &b2)        ...}

Grpc

Grpc is a Google-led RPC framework that uses the HTTP/2 protocol and PROTOBUF as a serialization tool. Its clients provide objective-c, Java interface, server side has Java, Golang, C + + and other interfaces. Using GRPC, you can easily invoke the methods of other processes, calling the data that needs to be transferred using proto encoding. This can effectively improve the coding efficiency and data transfer rate for large-scale projects.

Proto Service definition

An RPC service is a method that can be called remotely through parameters and return values, and we can simply interpret it as a function. Because GRPC is transmitted by encoding the data into protocal buffer. Therefore, we define the service method by Protocal buffers Interface Definitioin language (IDL), and also define the parameter and return value as the protocal buffer message type. The specific implementation is as follows, the file containing the following code is called Helloworld.proto:

syntax = "proto3";  package helloworld; // The greeter service definition.service Greeter {  // Sends a greeting  rpc SayHello (HelloRequest) returns (HelloReply) {}} // The request message containing the user's name.message HelloRequest {  string name = 1;} // The response message containing the greetingsmessage HelloReply {  string message = 1;}

Then, based on the service defined above, we can use protocal buffer compiler, the PROTOC, to generate the corresponding Golang code for the server side and the client. The generated code contains the methods that the client can make RPC and the interfaces that the server side needs to implement.

Assuming the current directory is $gopath/src/helloworld/helloworld, we will generate the GRPC corresponding Golang code with the following command:

protoc --go_out=plugins=grpc:. helloworld.proto

At this point, the Helloworld.pb.go file is generated under the directory

Server

Server.go

package main // server.go import (    "log"    "net"     "golang.org/x/net/context"    "google.golang.org/grpc"    pb "helloworld/helloworld") const (    port = ":50051") type server struct {}// 当接收到请求的时候回调用该方法// 参数由grpc自己根据请求进行构造 func (s *server) SayHello(ctx context.Context, in *pb.HelloRequest) (*pb.HelloReply, error) {    return &pb.HelloReply{Message: "Hello " + in.Name}, nil} func main() {    lis, err := net.Listen("tcp", port)    if err != nil {        log.Fatal("failed to listen: %v", err)    }    s := grpc.NewServer()    pb.RegisterGreeterServer(s, &server{})    s.Serve(lis)}

Where PB is the package of go files that we just generated based on Proto

Client

package main //client.go import (    "log"    "os"     "golang.org/x/net/context"    "google.golang.org/grpc"    pb "helloworld/helloworld") const (    address     = "localhost:50051"    defaultName = "world") func main() {    // 建立一个grpc连接    conn, err := grpc.Dial(address, grpc.WithInsecure())    if err != nil {        log.Fatal("did not connect: %v", err)    }    defer conn.Close()    // 新建一个客户端，方法为：NewXXXClinent(conn),XXX为你在proto定义的服务的名字    c := pb.NewGreeterClient(conn)     name := defaultName    if len(os.Args) >1 {        name = os.Args[1]    }    // 调用远程，并得到返回    r, err := c.SayHello(context.Background(), &pb.HelloRequest{Name: name})    if err != nil {        log.Fatal("could not greet: %v", err)    }    log.Printf("Greeting: %s", r.Message)}

RESTful turn GRPC

The advantages of using GRPC are many, the binary data can speed up the transfer speed, HTTP2-based multiplexing can reduce the number of connections between services, and the function of the same call method is also effective to improve the development efficiency. However, the use of GRPC will also face a problem, our micro-service must be to provide a restful interface, if the internal call using GRPC, in some cases to provide a function of the two sets of API interface, which not only reduces the development efficiency, but also increases the complexity of debugging. So I think there is a conversion mechanism, so that restful and GPRC can transform each other.

Grpc-gateway was born

Installation

First you have to install proto and Grpc according to the previous steps in this article, and then install some libraries as follows

go get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gatewaygo get -u github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swaggergo get -u github.com/golang/protobuf/protoc-gen-go

Usage

Define the proto file for the service

 syntax = "proto3"; package example; import "google/api/annotations.proto"; message StringMessage {   string value = 1; }  service YourService {   rpc Echo(StringMessage) returns (StringMessage) {     option (google.api.http) = {       post: "/v1/example/echo"       body: "*"     };   } }

option indicates which path requests are processed and how the request body (parameters) is handled, see Https://cloud.google.com/serv ...

Generate Go File

protoc -I/usr/local/include -I. \  -I$GOPATH/src \  -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \  --go_out=plugins=grpc:. \  path/to/your_service.protoprotoc -I/usr/local/include -I. \  -I$GOPATH/src \  -I$GOPATH/src/github.com/grpc-ecosystem/grpc-gateway/third_party/googleapis \  --grpc-gateway_out=logtostderr=true:. \  path/to/your_service.proto

The above generated two files, the first is the Pb.go file for Grpc server, and the second is the Pb.gw.go file, for Grpc-gateway, for GRPC and restful mutual transformation

Server

package mainimport (  "flag"  "net/http"  "github.com/golang/glog"  "golang.org/x/net/context"  "github.com/grpc-ecosystem/grpc-gateway/runtime"  "google.golang.org/grpc"      gw "path/to/your_service_package")var (  echoEndpoint = flag.String("echo_endpoint", "localhost:9090", "endpoint of YourService"))func run() error {  ctx := context.Background()  ctx, cancel := context.WithCancel(ctx)  defer cancel()  mux := runtime.NewServeMux()  opts := []grpc.DialOption{grpc.WithInsecure()}  err := gw.RegisterYourServiceHandlerFromEndpoint(ctx, mux, *echoEndpoint, opts)  if err != nil {    return err  }  return http.ListenAndServe(":8080", mux)}func main() {  flag.Parse()  defer glog.Flush()  if err := run(); err != nil {    glog.Fatal(err)  }}

Test

curl -X POST -k http://localhost:8080/v1/example/echo -d '{"name": " world"}{"message":"Hello  world"}

The

process is as follows: Curl uses post to send a request to the gateway, and the gateway acts as a proxy to convert the request through GRPC to Greeter_server,greeter_server and return the result via GRPC. After the gateway receives the results, it converts the JSON back to the front end.