Python Cookbook (3rd edition) Chinese version: 15.14 passing a Unicode string to the C function library

15.14 Pass a Unicode string to the C function library?

To write an extension, you need to pass a Python string to a library function in C, but this function does not know what to do with Unicode.

Solution?

There are a number of issues to consider here, but the main problem is that the existing C library does not understand the native Unicode representation of Python.
Therefore, your challenge is to convert the Python string into a form that can be understood by C.

For the purposes of the demonstration, there are two C functions below that manipulate string data and output it for debugging and testing.
A byte that uses form as a char *, int form,
And the other uses wchar_t *, int the form of a wide-character form:

voidPrint_chars(Char*S,IntLen){IntN=0;While(N<Len){Printf("%2x",(UnsignedChar)S[N]);N++;}Printf("\ n");}voidPrint_wchars(wchar_t*sint len) { Span class= "NB" >int n = 0while  (n < len ) {printf ( "%x  "s[nn++} printf ( "\n ");                /span>                

For byte-oriented functions print_chars() , you need to convert the Python string to a suitable encoding such as UTF-8.
Here is an example of such an extension function:

Static Pyobject *py_print_chars (Pyobject *self, Pyobject *args) {  char *s;  py_ssize_t  Len;  if (! Pyarg_parsetuple (args, "s#", &s, &len)) {    return NULL;  }  Print_chars (S, Len);  Py_return_none;}

For those library functions that need to handle machine-local wchar_t types, you can write extension code like this:

Static Pyobject *py_print_wchars (Pyobject *self, Pyobject *args) {  wchar_t *s;  py_ssize_t  Len;  if (! Pyarg_parsetuple (args, "u#", &s, &len)) {    return NULL;  }  Print_wchars (S,len);  Py_return_none;}

Here is an interactive session to demonstrate how this function works:

' Spicy jalape\u00f1o 'print_chars(s)-4a 6c, C3 B16fPRINT_WC HARs(s),4a 6c, F1 6f>>>. /c11> 

Look closely at how this byte-oriented function print_chars() accepts UTF-8 encoded data.
and print_wchars() how to accept Unicode encoded values

Discuss?

Before continuing with this section, you should first learn the features of the C function library that you access.
For many C libraries, it is often better to pass bytes rather than strings. To do this, use the following conversion code:

Static Pyobject *py_print_chars (Pyobject *self, Pyobject *args) {  char *s;  py_ssize_t  Len;  /* Accepts bytes, ByteArray, or other byte-like object *  /if (! Pyarg_parsetuple (args, "y#", &s, &len)) {    return NULL;  }  Print_chars (S, Len);  Py_return_none;}

If you still want to pass a string,
You need to know that Python 3 can be represented with a suitable string,
It does not directly map to char * wchar_t * the C function library that uses standard types or (more details refer to Pep 393).
Therefore, to represent this string data in C, some conversions are required.
The PyArg_ParseTuple() use of "s#" and "u#" format codes in can safely perform such conversions.

However, one drawback of this conversion is that it may cause the size of the original string object to increase.
Once converted, a copy of the transformation data is appended to the original string object, which can then be reused.
You can observe this effect:

>>>ImportSys>>>S=' Spicy Jalape\u00f1O>>>Sys.Getsizeof (s) 87>>> print_chars (s) 53 4a 6c C3 B1 6f>>> sys. Getsizeof (s) 103>>> span class= "n" >print_wchars (s) 53 4a 61 6c F1 6f>>> sys. Getsizeof (s) 163>>>   

For a small number of string objects, there may be no effect,
But if you need to handle a lot of text in the extension, you might want to avoid this loss.
The following is a revised version to avoid this memory loss:

Static Pyobject *py_print_chars (Pyobject *self, Pyobject *args) {  pyobject *obj, *bytes;  char *s;  py_ssize_t   Len;  if (! Pyarg_parsetuple (args, "U", &obj)) {    return NULL;  }  bytes = pyunicode_asutf8string (obj);  Pybytes_asstringandsize (Bytes, &s, &len);  Print_chars (S, Len);  Py_decref (bytes);  Py_return_none;}

wchar_tIt's even more difficult to avoid memory loss when it comes to processing.
Internally, Python uses the most efficient representation to store strings.
For example, a string containing only ASCII is stored as a byte array.
Strings that contain characters ranging from u+0000 to u+ffff use a double-byte representation.
Since the representation of the data is not single, you cannot convert the internal array to wchar_t * expect it to work correctly.
You should create an wchar_t array and copy the text to it.
PyArg_ParseTuple()The "u#" format code can help you to do it efficiently (it attaches the copy result to the string object).

If you want to avoid long memory loss, your only option is to copy the Unicode data. A temporary array,
Pass it to the C function, and then reclaim the memory of the array. The following is a possible implementation:

Static Pyobject *py_print_wchars (Pyobject *self, Pyobject *args) {  pyobject *obj;  wchar_t *s;  py_ssize_t Len;  if (! Pyarg_parsetuple (args, "U", &obj)) {    return NULL;  }  if ((s = pyunicode_aswidecharstring (obj, &len)) = = null) {    return null;  }  Print_wchars (S, Len);  Pymem_free (s);  Py_return_none;}

In this implementation, PyUnicode_AsWideCharString() create a temporary wchar_t buffer and copy the data in.
This buffer is passed to C and then released.
But when I write this book, there may be a bug here, followed by a Python question page.

If you know that the byte encoding required by the C function library is not UTF-8,
You can force Python to use the extension code to perform the correct conversion, as follows:

Static Pyobject *py_print_chars (Pyobject *self, Pyobject *args) {  char *s = 0;  int   Len;  if (! Pyarg_parsetuple (args, "es#", "Encoding-name", &s, &len)) {    return NULL;  }  Print_chars (S, Len);  Pymem_free (s);  Py_return_none;}

Finally, if you want to work directly with Unicode strings, here is an example that shows the underlying operation access:

Static Pyobject *py_print_wchars (Pyobject *self, Pyobject *args) {  pyobject *obj;  int n, Len;  int kind;  void *data;  if (! Pyarg_parsetuple (args, "U", &obj)) {    return NULL;  }  if (Pyunicode_ready (obj) < 0) {    return NULL;  }  Len = pyunicode_get_length (obj);  Kind = Pyunicode_kind (obj);  data = Pyunicode_data (obj);  for (n = 0; n < len; n++) {    Py_ucs4 ch = pyunicode_read (kind, data, n);    printf ("%x", ch);  }  printf ("\ n");  Py_return_none;}

In this code, PyUnicode_KIND() andPyUnicode_DATA()
These two macros are related to the variable-width storage of Unicode, which is described in Pep 393.
kindThe variable encodes the underlying storage (8-bit, 16-bit, or 32-bit) and information about the cached data pointer.
In fact, you don't need to know anything about these values,
You only need to pass them to the macro when you extract the characters PyUnicode_READ() .

And the last few words: When passing a Unicode string from Python to C, you should try to be as simple as possible.
If you have both UTF-8 and wide characters, select UTF-8.
Support for the UTF-8 is more common, it is not easy to make mistakes, and the interpreter can support it better.
Finally, make sure you read the relevant documentation on processing Unicode carefully.

Albert (http://www.aibbt.com/) The first artificial intelligence portal in China

Python Cookbook (3rd edition) Chinese version: 15.14 passing a Unicode string to the C function library