Basic description and use of Protocol Buffers, protocolbuffers

Basic description and use of Protocol Buffers, protocolbuffers

The reason we need to use protobuf is that when processing the contest module of OJ, we encounter a problem that a lot of information needs to be stored when generating contestRank. If we use the model for storage, the modification will be very troublesome if we continue to add other information in the future, and the implementation will be complicated, because for rank, the primary key of each rank is UserID first, and the basic information stored includes the number of AC, the question AC, and the penalty. The question AC includes the question ID as the primary key, attribute (whether it is AC, AC time, penalty, submit statistics for this question), and the submit statistics for each question is a submit entity, so that there will be multi-level nesting. If more information is added later. Another reason is that storage is not convenient, and the logic concept is not very clear. You need to save various information to make it complex.

The main mechanism of protobuf is to serialize the stored information into a string for storage. When information is required, the original information can be restored through deserialization, and the multi-level nested attributes can be implemented. So try to use this to solve rank storage (/□ \)

1: What is protobuf?

Well, this is an encyclopedia. see: protocolbuffer is a data exchange format of google. It is language-independent and platform-independent. Google provides implementation in three languages: java, c ++, and python. Each implementation includes the compiler and library files of the corresponding language. Because it is a binary format, it is much faster than using xml for data exchange. It can be used for data communication between distributed applications or data exchange in heterogeneous environments. As a binary data transmission format with excellent efficiency and compatibility, it can be used in many fields such as network transmission, configuration file, and data storage.

Simply put, the information of a certain data structure is saved in a certain format. It is mainly used for data storage and transmission protocol formats.

2: download and install:

Address: http://code.google.com/p/protobuf/downloads/list

Installation:

 tar -xzf protobuf-2.1.0.tar.gz  cd protobuf-2.1.0  ./configure   make  make check  make install

3: Create

There are various examples on the Internet, such as the writing of simple books.

First, we need to write a proto file to define the structured data to be processed in our program. In protobuf terminology, the structure

Data is called Message. For example:

package contest_submit; message ContestSubmit {   required int32 id = 1;   required int64 timestamp = 2; }

In the preceding example, the id is int32 and the timestamp is int64. Required indicates that this is required, a bit like not null in the database, and

Type is optional, which is optional.

To compile the. proto file, run the following command:

protoc -I=./ --python_out=./ ./*.proto

Indicates that the input. proto file is in the current directory, and the generated compilation file is output to the current directory. The source file is all files ending with. proto.

. The complete online format is:

If your proto file is stored under $ SRC_DIR and you want to put the generated file in the same directory, run the following command: protoc-I = $ SRC_DIR -- python_out = $ DST_DIR $ SRC_DIR/file description. proto

After compilation, the contest_submit_pb2.py file is generated (for the above example)

Of course, protobuf can also be nested. You can declare the objects of another class in one class, for example, the implementation of rank:

package contest_rank; message Submit {   optional string status = 1;   optional string date_time = 2;   optional string runID = 3; }  message Problem {   optional int32 problemID = 1;   repeated Submit submit = 2;   optional int32 acindex = 3;   optional int32 totalindex = 4;   optional int32 time = 5;   optional int32 FB = 6; }  message Rank {   optional string userID = 1;   optional string contestID = 2;   repeated Problem problem = 3;   optional int32 penalty = 4;   optional int32 ac = 5;   optional int32 total = 6; }  message ContestRankList {   optional string contestID = 1;   repeated Rank rank = 2; }

In this way, we can implement the previous requirement, namely, multi-level information nesting.

Usage:

First, remember to load the header file from the folder. proto import rank_pbs

When rank storage is implemented, a rank class is first declared for subsequent data processing.

class Rank():   class Problem():     class Submit():       def __init__(self, runID = "", status="", date_time=""):         self.status = status         self.date_time = date_time         self.runID = runID      def __init__(self, problemID):       self.problemID = problemID       self.submit_list = []       self.acindex = 0       self.totalindex = 0       self.time = 0       self.FB = 0      def add_submit(self, submit):       self.submit_list.append(submit)    def __init__(self, userID, contestID):     self.userID = userID     self.contestID = contestID     self.problem_list = {}     self.ac = 0     self.penalty = 0     self.total = 0 

When we get the rank_list (rank_list is a rank dictionary), we first declare a proto carrier:

contest_rank_list = rank_pb2.ContestRankList()

Then fill in the information. For example, add the competition ID, that is, contest_rank_list.contestID = contestID. Then, according to the protobuf structure

Enter the required information. For nested entities, the add () method can be used for each declaration. For example, in a single game, there will be multiple users.

So add information according to user ID:

rank_proto = contest_rank_list.rank.add()rank.load_data_to_proto(rank_proto)

Load_data_to_proto is a method of the rank class. The content is to add the information to protobuf:

def load_data_to_proto(self, rank):     rank.userID = self.userID     rank.contestID = self.contestID     rank.ac = self.ac     rank.penalty = self.penalty     rank.total = self.total      for problemID in self.problem_list:       problem = self.problem_list[problemID]       p = rank.problem.add()       p.problemID = problem.problemID       p.acindex = problem.acindex       p.totalindex = problem.totalindex       p.time = problem.time       p.FB = problem.FB       for submit in problem.submit_list:         s = p.submit.add()         s.status = submit.status         s.date_time = submit.date_time         s.runID = submit.runID     return rank 

In this way, we store the data in the declared protobuf, and serialize the data during storage. For example, we store the data in SSDB:

ssdb_api.SetContestRankListProto(contestID, contest_rank_list.SerializeToString())

So far, the information has been stored in the database, so the next step is how to read it:

For the rank information we store, that is:

 contest_rank_list = ssdb_api.GetContestRankListProto(contest_id) rank_list = rank_pb2.ContestRankList() rank_list.ParseFromString(contest_rank_list)

That is, declare a protobuf object again, and then deserialize it to get the information we need, simple (/□ \).

You can use the parent name and sub-name to access the required data.