Analysis on Chinese path supported by gdal C #

The gdal library has many problems with C # support, and one of them is the support for Chinese paths (the other is to obtain the coordinate information of the image through the OGR library ).

For C #'s support for Chinese paths, I should be familiar with my previous blogs. If I use a modified gdal library, you can set the following code to allow C # to directly support Chinese paths. If you use the official library, you can directly support the Chinese path without setting it.

// Register all drivers for OGR. registerall (); // to support Chinese paths, add the following code osgeo. gdal. gdal. setconfigoption ("gdal_filename_is_utf8", "yes"); // to support Chinese characters in the SHP Attribute Table fields, add the following osgeo statement. gdal. gdal. setconfigoption ("shape_encoding ","");

Yesterday, a friend said that he tested the C # version and found that the Chinese path can be used sometimes, sometimes not. Setting gdal_filename_is_utf8 does not help.

We found through tests today that,As long as the number of Chinese characters in the file name is an even number, there is no impact, reading and creation are normal, if the number of Chinese characters in the file name is an odd number, certainly cannot be read and created.

For example, the following file name is normal:

D: \ New Folder \ new 1. shpd: \ Miyun data \ 0.shp of line separation

But the following is definitely not true:

D: \ New Folder \ new 1. shpd: \ Alibaba Cloud data \ line separation 0.shp

The following describes how to debug the gdal library through the C # program and find the cause. According to the method of cross-language debugging in the previous blog, set a breakpoint for the open function in the C # program, start the debugging, and the program is interrupted here.

First, use a normal path that can be opened by the gdal library for testing, as shown in.

Press the F11 key to enter the swig encapsulated C # code, as shown in.

Here, we found such code.

  public static DataSourceOpen(string utf8_path, int update) {    IntPtr cPtr =OgrPINVOKE.Open(System.Text.Encoding.Default.GetString(System.Text.Encoding.UTF8.GetBytes(utf8_path)),update);    DataSource ret = (cPtr ==IntPtr.Zero) ? null : new DataSource(cPtr, true, ThisOwn_true());    if(OgrPINVOKE.SWIGPendingException.Pending) throwOgrPINVOKE.SWIGPendingException.Retrieve();    return ret;  }

When ogrpinvoke is called, the path is encoded and converted. The core code is as follows:

System.Text.Encoding.Default.GetString(System.Text.Encoding.UTF8.GetBytes(utf8_path))

As you can see from the code, swig first converts the C # default string to the default encoding using utf8 encoding. The above path "D: \ New Folder \ new 1. after this sentence is converted, the SHP becomes "D: \ slow board, slow board, and slow board. SHP ". This string is passed into the gdal library, in the file gdal-1.10.0 \ port \ cpl_vsil_win32.cpp function vsivirtualhandle * vsiwin32filesystemhandler: open (const char * pszfilename, const char * pszaccess) encoding conversion is performed again. As shown in.

If gdal_filename_is_utf8 = yes is set, the system first converts the encoding from utf8 to ucs2 encoding. After this sentence, we found that the path was programmed again, for example:

In this way, the gdal library can open the file normally. Next, let's look at a path that cannot be opened by gdal and repeat the above steps. Next, we will only take key locations.

The first step is to set the breakpoint when opening, the file path is"D: \ New Folder \ new 1.shp"

Then the path passed in to the gdal library is changed to"D: \ slow board, please refer to the slow board, please ?. SHP". Then, it is called"D: \ create a folder \ create açá ?. SHP". As shown in.

As long as the question mark (?) appears in the path (?), There must be a problem with this path, whether it is garbled or not. So this path cannot be opened.

Through the above steps, we can determine that the C # path is easy to use, and there is a problem after the conversion through the encoding in swig, so we can think of it as a problem of encoding conversion.

In the swig encapsulated interface, use system. text. encoding. default. getstring (system. text. encoding. utf8.getbytes (utf8_path) for conversion. The following code snippet is a simple test code for verification.

Staticvoid main (string [] ARGs) {string strutf8 = "d :\\ New Folder \ new 1.shp"; byte [] byutf8 = system. text. encoding. utf8.getbytes (strutf8); string strdefault = system. text. encoding. default. getstring (byutf8); byte [] bydefault = system. text. encoding. default. getbytes (strdefault); string strutf8n = system. text. encoding. utf8.getstring (bydefault );}

First, let's look at a path that can be normally accessed by gdal. First, check the conversion before turning back. A comparison of three strings is made. For example, we can see that after converting to default and then to utf8, same as the original path. Therefore, the gdal database can be accessed normally.

The content of the byte array obtained before and after conversion is completely consistent, as shown in:

Next, we will use a path that cannot be accessed by gdal for testing. After the conversion is viewed, We will convert it back to compare the three strings. For example, we can see that after converting to default and then to utf8, it has changed with the original path.

The following compares the two converted byte arrays. It is reasonable to say that the byte arrays in the memory should be the same. The following compares the contents of the two byte arrays, as shown in, the 27 and 28 Before array conversion are 132 and 49 respectively. After conversion, the two bytes are changed to one byte (63 ).

We can see from here that the problem lies here. Convert the value in the ASCII code table into a string. The following figure is displayed. An English character occupies one byte, while a Chinese character occupies three bytes. During transcoding, two bytes are used as a group for transcoding. That is to say, for an even number of Chinese characters, converting to a byte is an even number three times, and the result must be an even number, therefore, transcoding based on two bytes can be completed. If the number of Chinese characters is an odd number, the number of bytes is a three-fold odd number, and the result must be an odd number. transcoding is performed based on two bytes, there will certainly be one more system. The question mark (?) may be used if the other system does not know each other (?) .

Therefore, it can be thought that it is normal for Chinese characters to be even numbers, and problems may occur for odd numbers, which is exactly the same as the result of gdal. The last section above is my personal analysis, which does not mean that Microsoft implements this internally. Maybe this may be counted as a bug in C? I wonder if Microsoft has found this problem.