Unicode agent programming using the Java language

The early Java version uses a 16-bit char data type to represent Unicode characters. This design method is sometimes reasonable because all Unicode characters have a value less than 65,535 (0xFFFF) and can be represented by 16 digits. However, Unicode later increased the maximum value to 1,114,111 (0X10FFFF). Because 16 bits are too small to represent all Unicode characters in Unicode version 3.1, 32-bit values-called code point-are used in UTF-32 encoding mode.

However, with a 32-bit value, the 16-bit memory usage is more efficient, so Unicode introduces a new design method to allow the use of 16-bit values. This design method used in UTF-16 allocates a value of 1,024 to 16-bit high surrogate, assigning an additional 1,024 value to the 16-bit low proxy (lower surrogate). It uses a high proxy plus a low proxy-one proxy pair (surrogate pair)-to represent 1,114,111 (0X10FFFF) values between 65,536 (0x10000) and 1,048,576 (0x100000) (1,024 and Product of 1,024).

Java 1.5 Preserves the behavior of the char type to represent the UTF-16 value (in order to be compatible with an existing program), and it implements the concept of code bit to represent the UTF-32 value. This extension (implemented according to the JSR 204:unicode supplementary Character Support) does not require remembering the exact value of Unicode code bits or conversion algorithms-but it is important to understand the correct usage of the proxy API.

East Asian countries and regions have increased the number of characters in their character sets in recent years to meet user needs. These standards include GB 18030 from China's National standards Organization and JIS X 0213 from Japan. Therefore, procedures that seek to comply with these standards are more necessary to support Unicode proxy pairs. This article explains the relevant Java APIs and coding options, and plans to redesign their software from characters that can only use char type to readers who can handle new versions of proxy pairs.

Sequential access

Sequential access is a basic operation that handles strings in the Java language. In this way, each character in the input string is accessed sequentially from beginning to end, or sometimes from the tail to the head. This section discusses the use of sequential access methods to create 7 technical examples of a 32-bit code-bit array from a string and estimates their processing time.

Example 1-1: Benchmark (does not support proxy pairs)

Listing 1 assigns the 16-bit char type value directly to the 32-bit code-bit value and does not consider the proxy pair at all:

Listing 1. Proxy pair is not supported

int[]　 toCodePointArray(String　str)　{　//　Example　1-1
　　int　len　 =　str.length();　　　　　//　the　length　of　str
　　int[]　acp 　=　new　int[len];　　　　//　an　array　of　code　points

　　for　(int　i　=　0,　j　=　0;　i　<　len;　i++)　{
　　　 　acp[j++]　=　str.charAt(i);
　　}
　　return　acp;
}

Although this example does not support proxy pairs, it provides a processing time datum to compare subsequent sequential access examples.

Example 1-2: Using Issurrogatepair ()

Listing 2 uses Issurrogatepair () to calculate the total number of agents. After counting, it allocates enough memory to store this value in a code-bit array. It then enters a sequential access loop, using Ishighsurrogate () and Islowsurrogate () to determine whether each agent is a high agent or a low proxy for the character. When it finds a high proxy with a low proxy, it uses Tocodepoint () to convert the proxy pair to a code bit value and increase the current index value by 2. Otherwise, it assigns this char type value directly to a code bit value and increases the current index value by 1. This example has a processing time of 1.38 times times longer than the example 1-1.

Listing 2. Limited support

int[]　 toCodePointArray(String　str)　{　//　Example　1-2　
　　int　len 　=　str.length();　　　　　//　the　length　of　str
　　int[]　 acp;　　　　　　　　　　　　//　an　array　of　code　points
　　 int　surrogatePairCount　=　0;　　　//　the　count　of　surrogate　 pairs

　　for　(int　i　=　1;　i　<　len;　i++)　{
　　　　if　(Character.isSurrogatePair(str.charAt(i　-　1),　 str.charAt(i)))　{
　　　　　　surrogatePairCount++;
　　　 　　　i++;
　　　　}
　　}
　　acp　=　new　int[len　- 　surrogatePairCount];
　　for　(int　i　=　0,　j　=　0;　i　< 　len;　i++)　{
　　　　char　ch0　=　str.charAt(i);　　　　　// 　the　current　char
　　　　if　(Character.isHighSurrogate(ch0) 　&&　i　+　1　<　len)　{
　　　　　　char　ch1　=　 str.charAt(i　+　1);　//　the　next　char
　　　　　　if　 (Character.isLowSurrogate(ch1))　{
　　　　　　　　acp[j++]　=　 Character.toCodePoint(ch0,　ch1);
　　　　　　　　i++;
　　 　　　　　　continue;
　　　　　　}
　　　　}
　　　　 acp[j++]　=　ch0;
　　}
　　return　acp;
}