Chinese processing in Ruby/rails (1)

Because the current Ruby does not support Unicode, Chinese-related processing has been vague in ruby, which is basically the path of their respective attempts. I also listed some ideas to help me sort out their own ideas.

Utf8 and Unicode

UTF is actually (UNICODE translation format), that is, the Unicode encoding during transmission, including utf8 and UTF16. In fact, uf8 is a very unfriendly encoding for Chinese characters. Why, in utf8, a Chinese character occupies 3 bytes, while an English ASCII character occupies only 1 byte. In this way, the Chinese character size expands greatly, on the other hand, we cannot simply calculate the number of characters by calculating the number of bytes. UTF16 is normal, and each character occupies two bytes. however, at present, UTF16 is rarely followed by UTF32, and utf8 is a widely accepted encoding.

Utf8 or gb2312/GBK

I personally chose utf8 because I think gb2312/GBK is not a common encoding. If you use it, there may be other problems (such as multi-language support)

Conversion Encoding

We need the require 'iconv' library and then execute

Iconv. Conv ("utf8", "GBK", "...") # convert a character from GBK to utf8

When the source character set contains invalid encoding, we need to inform iconv to ignore it. Otherwise, iconv reports an error and stops.

Iconv.conv("utf8//IGNORE","GBK//IGNORE","....")

Utf8 Chinese operations:

Count of characters:

In the default Ruby environment, you need to execute

require 'jcode'

$ Kcode = 'U' # Or $ kcode = 'utf8' are equivalent.

Count of characters:

Use jsize or jlength

"Hello". size #6

"Hello". jsize #2

Character Truncation

"Hello" [] # garbled

"Hello". Scan (/./) [0, 1]. Join # "You"

Of course, there is another way:

Using UTF16 encoding, each word is two bytes (the header has two bytes). We can first convert the character to UTF16, and then follow the [0, the N * 2 + 2] method intercepts the first n characters (test successful. verifying)

Iconv. Conv ("utf8", "UTF16", iconv. Conv ("UTF16", "utf8", "Hello, B in a") []) # Hello