Essential knowledge for Creating excellent programs: character encoding (2)-Unicode and character encoding that must be known by software developers

Unicode and character encoding required by software developers

This is an article translated from Joel Spolsky, "the absolute minimum every software developer absolutely, positively must know about Unicode and character sets", which is classic.

[Added readability during translation, with a few changes]

Original article: http://www.joelonsoftware.com/articles/Unicode.html

Have you ever been confused by the Content-Type tag in HTML files? For example:

<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />

I often see it, but I don't know what it is used?

Have you received an email from a foreign country, such as Bulgaria? The title cannot be properly displayed and changed "???? ?????? ??? ???? "?

Sadly, I found many software developers do not know much about this mysterious world: character sets, character encoding, Unicode, and other similar things. Two years ago, a fogbugz beta tester wanted to know if the software could process emails from Japan written in Japanese. [Mark, I am not sure what it means here]. When looking at the e-mail content we used to parse mime, I found that it was completely incorrectly converted into character encoding, so I can only write a piece of code to restore the wrong conversions and operate character encoding in the correct way.

When I see another commercial function library, that's right. The implementation of character encoding is also bad. I contacted its developer and he said there was no solution. Like many programmers, he also hopes that those problems will "disappear by themselves.

It is impossible to let the problem "disappear by yourself. When I discovered that the popular web development tool PHP has almostcomplete ignorance of character encoding issues, blithely using 8 bits for characters, making
It darn near impossible to develop good international web applications, I thought,Enough is enough.

So I have a message to tell you: if you are a programmer and you don't seem to understand the basic content of computer characters, character sets, encoding, and Unicode. Then I caught you. I swear to let you kneel in the submarine and punish you for 6 months of scallions in a row.

I want to tell you that these seemingly advanced things are not that difficult to understand.

This article will show you the most basic content that every programmer should understand. All previous information about "plain text = ASCII encoded strings = each character is a byte string" ("plain text = ASCII = characters are 8 bits ") it is not only wrong, but also hopeless. If you still write programs in this way, it's just like a doctor doesn't believe that bacteria exist. Therefore, do not write programs before reading this article.

Before we start, I should remind you that if you are a few people who know about internationalization, you may think the content of the following discussion is too simple. I try to minimize the threshold so that everyone can understand the general principle and write code that can manipulate any language, not just an English string. I should also tell you that character-related technologies are only a small part of the general software for international creation, because I can only write one thing at a time, so today I wrote about character sets and character encoding.

Starting from history

The simplest way to understand them is to describe its development in chronological order.

But now we will not talk about ebcdic, because it is too old to talk about that history.

Back in the semi-olden days, when UNIX first appeared, when K & R was still writing the C programming language, everything was so simple. Ebcdic is about to disappear. At that time, the computer could only represent English characters that do not contain the variant symbol (note: some European languages have the variant symbol at the top of the letter). We call it ASCII. In the ASCII code table, all characters are carried from the 32-to 127 characters. The space is 32, the letter "A" is 65, and so on. Therefore, these characters can be conveniently stored in 7 binary bits. At that time, most computers use eight binary digits as a byte. Therefore, not only can all ASCII characters be stored in one byte, but also one binary space. If it is a bit bad, you can use it for other evil purposes: Once there was a word processing software WordStar to fill that gap to mark the end of the word.
The characters below 32 are non-printable characters. They are used to make you complain. Haha, it's just a joke. They are actually used as control characters. For example, 7 can make the computer beep, and 12 can exit the printed page and then load the new page.

Suppose you are using American English. Everything looks good.

Because there is still a lot of space left in a byte, many people will think of "oh, we can leave 128 to 255 for our own use ". The problem is that many people share the same idea. They fill in their content between 128 and 255 as they wish.

The IBM-PC had something that came to be known as the OEM character set which provided some accented characters for European classes ages anda bunch of Line Drawing characters... horizontal bars, vertical
Bars, horizontal bars with little dingle-dangles dangling off the right side, Etc ., and you cocould use these line drawing characters to make spiffy boxes and lines on the screen, which you can still see running on the 8088 computer at your dry cleaners '. in
Fact as soon as people started buying PCs outside of America all kinds of different OEM character sets were dreamed up, which all used the top 128 characters for their own purposes. for example on some PCs the character code 130 wocould display as é, but on
Computers sold in Israel it was the Hebrew letter gimel (), so when Americans wocould send their r é sum és to Israel they wowould arrive
Rsums.
In our cases, such as Russian, there were lots of different ideas of what to do with the upper-128 characters, so you couldn't even reliably interchange Russian documents.

Eventually this OEM free-for-all got codified in the ANSI standard. in the ANSI standard, everybody agreed on what to do below 128, which was pretty much the same as ASCII, but there were lots of different ways to handle the characters from 128 and on up,
Depending on where you lived. These different systems were calledCode pages. So for example in Israel DOs used a code page called 862, while Greek users used 737. They were the same
Below 128 but different from 128 up, where all the funny letters resided. the national versions of MS-DOS had dozens of these code pages, handling everything from English to Icelandic and they even had a few "multilingual" code pages that cocould do Esperanto
And GalicianOn the same computer! Wow!But getting, say, Hebrew and Greek on the same computer was a complete
Impossibility unless you wrote your own custom program that displayed everything using bitmapped graphics, because Hebrew and Greek required different code pages with different interpretations of the high numbers.

Meanwhile, in Asia, even more crazy things were going on to take into account the fact that Asian alphabets have thousands of letters, which were never going to fit into 8 bits. this was usually solved by the messy system called DBCS, the "Double Byte Character
Set "in whichSomeLetters were stored in one byte and others took two. it was easy to move forward in a string, but dang near impossible to move backwards. programmers were encouraged not to use s ++ and S -- to move backwards and forwards, but instead
To call functions such as Windows 'ansinext and ansiprev which knew how to deal with the whole mess.

But still, most people just pretended that a byte was a character and a character was 8 bits and as long as you never moved a string from one computer to another, or spoke more than one language, it wocould sort of always work. but of course, as soon as
Internet happened, it became quite commonplace to move strings from one computer to another, and the whole mess came tumbling down. Luckily, Unicode had been pinned Ted.