Google protocol buffer uses UTF-8 in Python

Google protocol buffer is easy to use, but there are some minor problems in Python. For example, UTF-8 only supports Unicode strings. In our system, both the storage and transmission are UTF-8. Therefore, in code (C ++, Java, and Python), the UTF-8 format is used in a unified manner. Therefore, it is inconvenient. For example, C ++ needs to convert from UTF-8 to Unicode and parse it in Python. In fact, Google protocol buffer for python is not transcoded.

The problem I encountered was: C ++ reads files (UTF-8 in the file), and then Python accepts them. c ++ sends UTF-8, but Python requires Unicode, very depressing.

The description of the string type is as follows: UTF-8 and ASCII are supported. In fact, the Python version requires Unicode, which is really ridiculous. It can be seen that it is better than anything else.

String

A string must always contain UTF-8 encoded or 7-bit ASCII text.

String

String

Download the source code of Google protocol buffer, such as http://code.google.com/p/protobuf/downloads/list, and download 2.0.3133.
Go to/protobuf-2.0.3/Python/Google/protobuf/internal directory
Encode. py

1. Def appendstring (self, field_number, value ):
2. "Appends a length-prefixed Unicode string, encoded as UTF-8 to our buffer,
3. With the length varint-encoded.
4. """
5. # Self. appendbytes (field_number, value. encode ('utf-8 '))
6. # Remove value. encode. I used to be UTF-8. Why do I still encode it?
7. Self. appendbytes (field_number, value)

Decode. py

1. Def readstring (Self ):
2. "Reads and returns a length-delimited string ."""
3. Bytes = self. readbytes ()
4. # Return Unicode (bytes, 'utf-8 ')
5. # Change it to the following sentence and do nothing. If it is an encoding, it returns an encoding.
6. Return bytes

Type_check.py

1. Class unicodevaluechecker (object ):
3. "Checker used for string fields ."""
5. Def checkvalue (self, proposed_value ):
6. If not isinstance (proposed_value, (STR, Unicode )):
7. Message = ('%. 1024r has type % s, but expected one of: % s' %
8. (Proposed_value, type (proposed_value), (STR, Unicode )))
9. Raise typeerror (Message)
11. # If the value is of Type 'str' make sure that it is in 7-bit ASCII
12. # Encoding.
13. # Inexplicably, why try convert to Unicode and convert it from ASCII to comment out
14. # If isinstance (proposed_value, STR ):
15. # Try:
16. # Unicode (proposed_value, 'ascii ')
17. # Couldn't unicodedecodeerror:
18. # Raise valueerror ('%. 1024r isn/' t in 7-bit ASCII encoding .'
19. # % (Proposed_value ))

After the change, re-install it and it will not convert the code to you. The encoding you want to transfer is what encoding. I have used C ++, and it seems that I will not handle the encoding problem. However, it seems that bytes in the proto data type can also meet the requirements. bytes does not process any data.