The Unicode puzzle for JavaScript

in other words, JavaScript is a bit weird when it comes to Unicode. This article explains the painful parts of JS on Unicode, then provides a solution and explains how to improve these issues in future ECMAScript6.

Unicode Basics
to get a better understanding of Unicode issues in JavaScript, make sure you understand what Unicode is.
The simplest we can think of Unicode as a database, any symbol you can think of that corresponds to a number (we call this number its code bit) and a unique name. In this way, we can easily reference a symbol without having to use the symbol itself directly.

For example:

code bits are usually represented by 16 decimal digits, with a 0 complement, and a minimum of 4 digits plus a u+ prefix.

The Yards range from u+0000 to U+10FFFF. Can represent more than 1.1 million symbols. In order to organize such a large amount of data, Unicode divides the codes into 17 planes, with approximately 65,000 code bits per plane.

The first and most important part is called the Basic multilingual Plane or BMP, which contains the symbols we would normally use. In English text documents, it is often sufficient to use only BMP .

There are 1 million available code bits left in addition to BMP. The plane containing the 1 million code bits is called a supplemental plane or an interstellar plane.

The interstellar plane is very easy to identify: When you need to use a 16 binary number greater than 4 bits to represent the code bit, this code bit is the interstellar code bit.

Now that we know the Unicode basics, let's look at how it's applied to JavaScript strings.

Escape Sequences

You may have seen these things before:

1. >> ' \x41\x42\x43 '
2. ' ABC '
3. >> ' \x61\x62\x63 '
4. ' ABC '

These are called 16 binary escape sequences . They contain 2-bit, 16-digit code bits . For example, ' \x41 ' means u+0041 LATIN capital letters A. Careful readers may find that these escape sequences can represent Code bit from u+0000 to U+00FF.

There is also a common escape:

1. >> ' \u0041\u0042\u0043 '
2. ' ABC '
3. >> ' I \u2661 javascript! '
4. ' I? Javascript! '

These are called Unicode escape sequences . They use a 4-bit 16 binary number to represent a code bit . For example: ' \u2661 ' means u+2661 white heart SUIT. The range represented by these escape sequences is u+ 0000 to U+FFFF, contains all BMP.

So what about the other planes? Like the interstellar plane? We need more than 4 digits of 16 binary numbers to indicate their code bit ... How to escape??
in ECMAScript 6, this is simple because a new escape method has been added: Unicode code bit escaped.
For example:

1. >> ' \u{41}\u{42}\u{43} '
2. ' ABC '

1. >> ' \u{1f4a9} '
2. '? '//u+1f4a9 PILE of POO

for backwards compatibility with ECMASCRIPT5 and earlier environments, a bad solution is to use an alternative combination:

1. >> ' \ud83d\udca9 '

1. '? '//u+1f4a9 PILE of POO

a star symbol is made up of both. Note that these two components have lost their own code-point meaning.

with this alternative combination, all the interstellar code bits can be expressed: you should have sensed that a single code position can be expressed in a BMP that is confused with the interstellar symbols that need to be represented by a substitution combination, confusing, and even causing nasty consequences.

calculate the number of characters in JavaScript
What would you do if you wanted to calculate the length of the string?
my first thought was to use the length property.

1. >> ' a '. Length//u+0041 LATIN Capital Letter A
2. 1
4. >> ' A ' = = ' \u0041 '
5. True
7. >> ' B '. Length//u+0042 LATIN Capital Letter B
8. 1
10. >> ' B ' = = ' \u0042 '
11. True

In the above example, the Length property does represent the number of characters. (This makes sense, because if we use escape sequences to represent this character, we just need an escape (\u0041).)


look at a different example:

1. >> '? '. Length//u+1d400 mathematical BOLD Capital A
2. 2
3. >> '? ' = = ' \ud835\udc00 '
4. True
5. >> '? '. Length//u+1d401 mathematical BOLD Capital B
6. 2
7. >> '? ' = = ' \ud835\udc01 '
8. True
9. >> '? '. Length//u+1f4a9 PILE of POO
10. 2
11. >> '? ' = = ' \ud83d\udca9 '
12. True

inside JavaScript, use the alternative combination mentioned above to represent interstellar characters and expose 2 characters that make up an alternate combination. If you use ECMAScript 5-compatible escape sequences to represent symbols, you need 2 escape characters to represent an interstellar symbol. This is confusing. , because people usually consider them as a whole of a Unicode symbol or letter, rather than thinking of an interstellar character as 2 parts.

The Unicode puzzle for JavaScript