PROTOBUF Study Notes

The article is from the official Google document translation, see the original language guide. Part of the content may be repeated, hope many forgive me.

Suppose you want to define a "search request" message format, with each request containing a query string, the number of pages in which you are interested in the results of the query, and how many query results per page. You can define the. proto file for the message type in the following ways:

syntax = "proto3";message SearchRequest {  string query = 1;  int32 page_number = 2;  int32 result_per_page = 3;}

The SEARCHREQUEST message format has 3 fields, and the data that is hosted in the message corresponds to each of the fields. Each of these fields has a name and a type.

Attention:

• The first line of the file specifies that you are using the PROTO3 syntax: If you do not specify this, the compiler will use Proto2. This specified syntax line must be the first line of a non-empty non-annotated file.

Message definition

1. Specify the field type

In the example above, all the fields are scalar types: two integers ( page_number and result_per_page ), one string type (query). Of course, you can also specify other composition types for the field, including enumerations (enumerations) or other message types.

2. Assigning identification numbers

As in the file format above, each field has a unique numeric identifier in the message definition. These identifiers are used to identify each field in the binary format of the message and cannot be changed once it is started.

Note: the identification number within [1,15] occupies one byte when encoded. The identification number within [16,2047] occupies 2 bytes. Therefore, the identification number within [1,15] should be retained for those message elements that occur frequently. Remember: To reserve some identification numbers for frequently occurring identification numbers that may be added in the future.

The smallest identification number can start at 1, maximum to 2^29-1, or 536,870,911. You cannot use the [19000-19999] identification number, which is reserved in the PROTOBUF protocol implementation. If you do not want to use these reservation identifiers in the. proto file, you will be alerted at compile time.

3. Specify field Rules

The message field modifier specified must be one of the following:

• Required: A well-formed message must contain 1 of these fields. Indicates that the value must be set;
• Optional: The field can have 0 or 1 values (no more than 1) in a message format.
• Repeated: In a well-formed message, this field can be repeated any number of times (including 0 times). The order of duplicate values is preserved. Indicates that the value can be duplicated, which is equivalent to list in Java.

For some historical reasons, the repeated fields of the base numeric type are not encoded as efficiently as possible. In the new code, users should use special options [packed=true] to ensure more efficient coding. Such as:

repeated int32 samples = 4 [packed=true];

Required is permanent: when you identify a field as required, you should be very careful. If, in some cases, you do not want to write to or send a required field, change the original field modifier to optional you may encounter problems-the old version of the consumer will assume that the message without the field is incomplete, which may result in a non-purposeful rejection resolution. In this case, you should consider writing a custom message validation function specifically for the application.

Some of Google's engineers have come to the conclusion that using required does more harm than better; they prefer to use optional and repeated rather than required. Of course, this view is not universal.

4. Add more message types

You can define multiple message types in a. proto file. This is especially useful when defining multiple related messages-for example, if you want to define a reply message format that corresponds to the SearchResponse message type, you can add it to the same. proto file, such as:

message SearchRequest {  string query = 1;  int32 page_number = 2;  int32 result_per_page = 3;}message SearchResponse { ...}

5. Add comments

To add comments to the. proto file, you can use the C/c++/java-style double slash (//) syntax format, such as:

message SearchRequest {  string query = 1;  int32 page_number = 2;  // Which page number do we want?  int32 result_per_page = 3;  // Number of results to return per page.}

6. What did the. proto file generate

When the. proto file is run with the Protocolbuffer compiler, the compiler generates code for the selected language that can manipulate the message types defined in the. proto file, including getting, setting field values, serializing the message into an output stream, and parsing messages from one input stream.

• For go, the compiler generates a. pd.go file in bits per message type.
• For C + +, the compiler generates an. h file and a. cc file for each. proto file, with a corresponding class for each message in the. proto file.
• It's a little bit different for python-- The Python compiler generates a module with a static descriptor for each message type in the. proto file, which is used with a meta class (Metaclass) at runtime (runtime) to create the required Python data access class.
• For objective-c, the compiler generates a pbobjc.h file and PBOBJCM file for each message type, and each message in the. proto file has a corresponding class.

7. Scalar numeric types

A scalar message field can be automatically generated by accessing the type defined in the class.

For more information on how to encode various types of serialization messages, see here

8. Fields and default values for optional

An element in the message description can be marked as "optional" (optional). A well-formed message can contain 0 or one optional elements. When parsing a message, if it does not contain the element value of optional, then the corresponding field in the parsed object is set to the default value. The default value can be specified in the message description file. For example, to specify a default value of 10 for the Result_per_page field of a searchrequest message, the message format is defined as follows:

optional int32 result_per_page = 3 [default = 10];

If no default value is specified for the optional element, the default value associated with the specific type is used: for string, the default value is an empty string. for bool, the default value is False. For numeric types, the default value is 0. For enumerations, the default value is the first value in the enumeration type definition.

9. Enumeration

When you need to define a message type, you might want to specify a value in a predefined sequence of values for a field. For example, suppose you want to add a corpus field for each searchrequest message, and the value of corpus may be one of universal,web,images,local,news,products or video.

This can be done easily: by adding an enumeration (enum) to the message definition. A field of an enum type can be used only as a value in a specified set of constants (if an attempt is made to specify a different value, the parser treats it as an unknown field).

In the following example, an enumeration type called corpus is added to the message format-it contains all possible values-and a field of type corpus:

message SearchRequest {  string query = 1;  int32 page_number = 2;  int32 result_per_page = 3;  enum Corpus {    UNIVERSAL = 0;    WEB = 1;    IMAGES = 2;    LOCAL = 3;    NEWS = 4;    PRODUCTS = 5;    VIDEO = 6;  }  Corpus corpus = 4;}

If you define an alias for an enumeration constant, you need to set Allow_alias option to True, otherwise the protocol compiler generates an error message.

enum EnumAllowingAlias {  option allow_alias = true;  UNKNOWN = 0;  STARTED = 1;  RUNNING = 1;}enum EnumNotAllowingAlias {  UNKNOWN = 0;  STARTED = 1;  // RUNNING = 1;  // Uncommenting this line will cause a compile error inside Google and a warning message outside.}

Enumeration constants must be within the range of 32-bit integer values. Because the enum value is variable encoded and not efficient for negative numbers, it is not recommended to use negative numbers in an enum.

As shown in the previous example, you can define enumerations inside or outside a message definition-these enumerations can be reused in any message definition in a. proto file. Of course, you can declare an enumeration type in one message and use it in a different message-in the syntax format of Messagetype.enumtype.

When you run the protocol buffer compiler against an. proto file that uses an enumeration, the generated code will have a corresponding enum (for Java or C + +), or a special Enumdescriptor class (for Python). It is used to create a series of integer-Value symbolic constants (symbolic constants) in the class generated at run time.

Use

The message supports nested use as a field type in another message

message SearchResponse {    repeated Result results = 1;}message Result {    string url = 1;    string title = 2;    repeated string snippets = 3;}

Importing definition (Import)

You can import a type that is declared in a different profile by using an import statement

import "others.proto";

Protocol the buffer compiler looks for the -I / --proto_path imported file in the directory specified by the parameter, and if it is not specified, it is found by default in the current directory.

Message nesting

Chestnuts:

message SearchResponse {    message Result {        string url = 1;        string title = 2;        repeated string snippets = 3;    }    repeated Result results = 1;}

Internally declared message type names can be used only internally, and external references require a pre-parent message name, such as Parent.Type :

message SomeOtherMessage {    SearchResponse.Result result = 1;}

Multilayer Nesting Support:

message Outer {                // Level 0    message MiddleAA {         // Level 1        message Inner {        // Level 2            int64 ival = 1;            bool  booly = 2;        }    }    message MiddleBB {         // Level 1        message Inner {        // Level 2            int32 ival = 1;            bool  booly = 2;        }    }}

Map type

Proto3 supports map type declarations:

map<key_type, value_type> map_field = N;message Project {...}map<string, Project> projects = 1;

• The key, value type can be a built-in scalar type, or it can be a custom message type
• field does not support repeated property
• Do not rely on the map type's field order

Package (Packages)

Use the package name in the. proto file to avoid naming conflicts.

syntax = "proto3";package foo.bar;message Open {...}

In other message format definitions, the type can be used in the same way as the package name + message name, such as:

message Foo {    ...    foo.bar.Open open = 1;    ...}

In different languages, the package name definition has different effects on the code that is generated after compilation:

• C + +: corresponds to a C + + namespace, for example open in namespace Foo::bar
• In Java: The package is used as the Java pack name unless the option jave_package options are specified
• Python: The package is ignored
• Go: The package name is used by default unless option Go_package is specified
• In Javanano: with Java
• C #: The package is converted to a camel-like namespace, such as foo.bar, unless option csharp_namespace is specified

Defining Services (Service)

If you want to use the message type in an RPC (remote method call) system, you can define an RPC service interface in the. proto file, and the protocol buffer compiler generates the service interface code based on the different languages you choose.

For example, to define an RPC service and have a method that receives SearchRequest and returns a SearchResponse, you can define this in the. proto file as follows:

service SearchService {    rpc Search (SearchRequest) returns (SearchResponse) {}}

The generated interface code acts as a contract between the client and the server, and the server must implement all of the interface methods defined, and the client directly invokes a method of the same name to initiate a request to the server.

Compared to the egg, it is necessary to specify a request message even if the parameter is not required on the business, an empty message is generally defined.

Option (Options)

.protoYou can annotate a series of options when defining a file. The options do not change the meaning of the entire file statement, but can affect how it is handled in a particular environment. Full options are available to check with Google.

Some options are file-level, meaning that it can be used for the top-level scope and is not contained within any message, enum, or service definition. Some options are message-level and can be used inside the message definition. Of course, some options can be used in fields, enum types, enum values, service types, and service methods.

But so far, there is no valid option to work with these types.

Here are some common options:

• Java_package (File option): Specifies the package in which the Java class is generated, and if there is no explicit declaration java_package in the. proto file, the default package name is used. No need to generate Java code when not working
• Java_outer_classname (File option): Specifies the name of the generated Java class, if Java_outer_classname is not explicitly declared in the. proto file, The generated class name will be generated according to the name of the. proto file, in camel-type naming mode. such as (Foo_bar.proto generated Java class named Foobar.java), do not need to generate Java code does not play any role
• Objc_class_prefix (File option): Specifies the Objective-c class prefix, preceded by all classes and enumerated type names. Without a default value, you should use 3-5 uppercase letters. Note that all 2-letter prefixes are reserved by Apple.

Basic specifications

• The description file is a file suffix of. Proto, and statements other than the structure definition end with a semicolon

• The structure definition includes:message、service、enum

• RPC method definition end of semicolon optional

• The name of the message is named by the hump, and the field name is separated by lowercase letters and dashes.

message SongServerRequest {    required string song_name = 1;}

• Enums type name using Hump naming method, field naming by uppercase and underline

enum Foo {    FIRST_VALUE = 1;    SECOND_VALUE = 2;}

• Service and RPC method names are used in a camel-named style

• The message corresponds to the struct in Golang, and after compiling the Go code, the field name is converted to Camel

Compile

Generate Java, Python, C + +, Go, Ruby, Javanano, Objective-c, or C # code by defining a. proto file, and you need to install the compiler Protoc.

Refer to the GitHub project Google/protobuf install the compiler, the go language needs to install a special plug-in at the same time: Golang/protobuf.

To run the command:

protoc --proto_path=IMPORT_PATH --cpp_out=DST_DIR --java_out=DST_DIR --python_out=DST_DIR --go_out=DST_DIR --ruby_out=DST_DIR --javanano_out=DST_DIR --objc_out=DST_DIR --csharp_out=DST_DIR path/to/file.proto

Here is only a reference, the specific language of the compilation example, please refer to the detailed documentation,