Five Things Every programmer should know about Unicode: unicode programmers

Five Things Every programmer should know about Unicode: unicode programmers

Translator: Xiao Feng, source: diannong net

Link: codeceo.com/article/5-things-every-programmer-know-about-unicode.html

Unicode is an incredible and useful standard that enables computers, smartphones, and smart watches around the world to display the same information in the same way. Unfortunately, its complexity makes it a gold mine for frauds and pranks.

Previous news about the Android app of the fake WhatsApp appeared to be provided by the same developer as the official app. Frauds prevent verification by including unicode non-output spaces in developer names. The maintenance staff of Play store noticed that hackers had cheated more than 1 million people.

Unicode is an incredible and useful standard that enables computers, smartphones, and smart watches around the world to display the same information in the same way. Unfortunately, its complexity makes it a gold mine for frauds and pranks. If giants like Google cannot defend against the basic problems caused by Unicode, this may be a seemingly failed battle for small companies. However, most of these problems are caused by vulnerabilities. The following are the five most important things that developers need to know about Unicode to prevent fraud.

1. Many Unicode points are invisible

Unicode has several zero-width code points, such as a zero-width connector (U + 200D) and a zero-width non-connector (U + 200C ), these tools connected with hyphens are all hint ). They have no visible effect on the screen appearance, but they affect string comparison, which is why the fake WhatApp can survive the missed check for so long. Most of these characters are in regular punctuation blocks (from U + 2000 to U + 206F ). In general, there is no reason to allow anyone to use the code points of this punctuation block in the identifier, so they are least likely to be filtered. However, some other special code beyond this range is invisible, such as the Mongolian vowel separator (U + 180E ).

In general, it is dangerous to use Unicode to compare the uniqueness constraints with simple strings. The potential solution is to restrict the character sets allowed by identifiers and any other data that may be abused by frauds. Unfortunately, this does not completely solve the problem.

2. Many code points look very similar

Unicode has to have many similar characters to cover all the symbols used by all written languages in the world, so that people cannot distinguish these characters, but there is no problem in computer differentiation. This problem leads to a large misuse of Mimic-an interesting tool that replaces common symbols used in software development with Unicode characters with similar appearances, such as colons and semicolons. It can cause confusion in code compilation tools and confuse developers.

The appearance of a symbol is similar to that of a simple prank. Although it is a bit strange to be called a homomorphic attack, these vulnerabilities may indeed cause serious security problems. In April 2017, a security researcher registered a domain name that is very similar to apple.com and even obtained an SSL Certificate by mixing letters from different character sets. All major browsers happily display SSL locks and list domain names as secure.

Similar to mixed and invisible characters, there is little reason to allow the use of mixed character set names, especially domain names, in identifiers. Most browsers have taken measures to punish domain names of mixed character sets and display them as hexadecimal Unicode values so that users will not be easily confused. If you want to display identifiers to users, such as in search results, consider some similar methods to prevent confusion. However, this is not a perfect solution. Some domain names, such as sap.com or chase.com, can be easily built from a single block of a non-Latin character set.

Unicode Consortium released a confusing character list that can be used to automatically check the stockade. On the other hand, if you are looking for a fast path to create obfuscation, see Shapecatcher-this is a wonderful tool that lists Unicode symbols that look similar to plotting.

3. Non-standard planning

Normalization is very important for identifiers such as user names. It can help users input values in different ways but process them in a consistent manner. A common method for canonical identifiers is to convert all content to lowercase letters. For example, ensure that JamesBond is the same as jamesbond.

Because there are so many similar characters and overlapping sets, different standardized policies may be applied to different languages or unicode processing libraries, which may potentially open up security risks, if normalization is completed in several places. In short, do not consider lowercase transformations to be the same in different parts of the application. Mikael Goldmann from Spotify wrote an awesome event analysis on this issue in 2013 after one of their users found a method of account hijacking. Attackers can register unicode variants of other users' usernames (such as bigbird ). Different levels of the application process words differently-allows users to register malicious accounts, but resets the password of the target account.

4. There is no relationship between the screen display length and memory size.

When using the basic Latin and most European character sets, the text space on the screen or paper is roughly proportional to the number of symbols, and roughly proportional to the size of the text memory. That is why EM and EN are popular unit lengths. However, with Unicode, any similar assumption will become dangerous. There are some cute symbols, such as Bismallah Ar-Rahman Ar-Raheem (U + FDFD). This single character is longer than most English words, therefore, it is easy to jump out of the hypothetical visual closure on the website. This means that any type of line feed or text interrupt Algorithms Based on string characters may be easily fooled. Most terminal programs require a fixed-width font, so if one of them is displayed, you will see the ending quotation marks in a completely wrong place.

One misuse of this is the zargo text generator, which adds garbage around text fragments, which occupies more vertical space.

Of course, the entire problem of hiding the code points makes the memory size irrelevant to the screen length, so it can be well suited to input fields that may be enough to blow up the database field. It is not enough to filter out non-visual characters to prevent problems, because there are many other examples that do not occupy space.

A combination of Latin characters (for example, U + 036B and U + 036C) occupies the space of the previous letter, in this way, you can write multiple lines of text in a single text line ('n' ͫ O ͬ 'generates N ͫ O ͬ ). The tone marks used to represent the ritual recitations of the Hebrew Bible can be infinitely stacked in the same visual space, which means they can be easily abused, this will cause a large amount of information to be encoded to occupy the content of a single character on the screen. Martin Kleppe encodes the Conway's Game of Life for the browser's tone mark. View the source code of the page.

5. Unicode is not just passive data

Some code points are designed to influence the display mode of printable characters, which means that you can copy and paste data, not just input processing commands. A common prank is to switch the text direction from right to left (U + 202E. For example, search Ninjas on Google Maps. The query string will actually flip the direction of the search term. Although the search field on the page shows "ninjas", it actually searches for "sajnin ".

This vulnerability is so popular that it even becomes XKCD.

Hybrid data and processing commands-code that can be effectively executed-are by no means a good idea, especially if users can directly input them. This is a big problem for any user input contained in the page display. Most Web developers know that they can delete HTML tags to clear user input. However, pay attention to the Unicode control characters in the input. This is a simple way to solve any form of dirty words or content filtering-you only need to flip the word backward and include overwrites from right to left at the beginning.

Editing from right to left may not embed malicious code, but if you are not careful, it may damage the content or flip pages. A common method to prevent this situation is to put the content provided by the user into the input field or text area, so that the processing command will not affect the rest of the page.

Another type of processing command that is particularly problematic for display is the font transform selector. To avoid creating separate code for each color variant of each emoticon, Unicode allows you to use a conversion selector to mix the basic symbol with the color. The white flag, the font selector, and the Rainbow usually generate a rainbow flag. However, not all transformations are valid. In January 2017, a bug in iOS unicode processing allowed the publisher to remotely crash the iPhone by sending specially crafted messages. The message contains a white flag, a font transform selector, and a zero. At this time, iOS CoreText will enter emergency mode-try to select the correct variant and cause the operating system to crash. This prank acts on direct information, group chat, and even sharing contact cards. This problem also affects iPad and even some MacBook computers. We are almost helpless.

Similar bugs occur every several years. In 2013, the bug of Arabic character processing-which may cause OSX and iOS to crash-surfaced. All these errors are deeply buried in the OS text processing module, so typical client application developers cannot block them at all.