Re-recognize Unicode and UTF8 encoding

Re-recognize Unicode and UTF8 encoding

Re-recognize Unicode and UTF8 encoding

Until today, to be exact, I just knew that UTF-8 encoding and Unicode encoding are not the same, there is a difference between the embarrassing
There is a certain connection between them, to see the difference between them:

UTF-8的长度是不一定的，有可能是1、2、3字节Unicode长度一定，2个字节（USC-2）UTF-8可以和Unicode互相转换

The relationship between Unicode and UTF8

Unicode (16 binary)	UTF-8 (binary)
0000-007f	0xxxxxxx
0080-07ff	110xxxxx 10xxxxxx
0800-ffff	1110xxxx 10xxxxxx 10xxxxxx

The table above has 2 meanings, the first obvious is the correspondence between Unicode and the UTF-8 character range, and one can see how Unicode and UTF-8 convert to each other:

First, the conversion from UTF-8 to Unicode.

UTF-8 encoded binary and the above 3 formats to match, matching to remove the fixed bit (the table in the non-X position), and then from right to left each 8-bit group, not enough 8 bits left not to lead, together enough 2 bytes of bits, this is the UTF-8 corresponding Unicode encoding. Take a look at some of the following examples:

The text encoding format in the above picture is UTF-8, you can see its 16 binary representation with Winhex

字符=> UTF-8  => UTF-8二进制=> 去掉固定位置凑够16位的二进制 => 16进制汉 => E6B189 => 11100110 10110001 10001001=> 01101100 01001001 => 6C49字 => E5AD97 => 11100101 10101101 10010111=> 0101101101010111 => 5B57#下面是在chrome命令行下面运行的结果'\u6C49'"汉"'\u5B57'"字"#到这里的话，从UTF-8转换到Unicode已经是一件非常容易的事了，看看转换的伪代码读取一个字节，11100110判断该UTF-8字符的格式，属于第三种，3个字节继续读取2个字节得到 11100101 10101101 10010111按照格式去掉固定位     1011011 01010111不够16位，左边补零    01011011 01010111  => 5B57

And look at the conversion from Unicode to UTF-8.

Talk about the problem.

Say the cause of today's problem, from the front-end input many words, UTF-8 format each word up to 30 bytes, so it will be in the front-end and background verification, JavaScript is Unicode encoding, the backend program is UTF-8 encoding, now the solution is this

Front

function utf8_bytes(str){var len = 0, unicode;for(var i = 0; i < str.length; i++){unicode = str.charCodeAt(i);if(unicode < 0x0080) {++len;} else if(unicode < 0x0800) {len += 2;} else if(unicode <= 0xFFFF) {len += 3;}else {throw "characters must be USC-2!!"}}return len;}#例子utf8_bytes('asdasdas')8utf8_bytes('yrt燕睿涛')12

Background

#对于GBK字符串$len = ceil(strlen(bin2hex(iconv('GBK', 'UTF-8', $word)))/2);#对于UTF8字符串$len = ceil(strlen(bin2hex($word))/2);

5/21/2015 8:21:53 PM

The copyright belongs to the author Iforever (luluyrt@163.com) all, without the author's consent to prohibit any form of reprint, reprinted article must be in the article page obvious location to the author and the original text connection, otherwise reserve the right to pursue legal responsibility.

4 Floor Fragrance White Lotus Vegetarian really

Unicode is a character set that defines the number corresponding to each character. , UTF-8, UTF-16, etc. are encoded in a format that defines how the "number of characters corresponds" is stored in binary mode.

Re: The man running

@ Fragrance of White Lotus is really, I think it is better to say:, Uincode define each number (0X0000~0XFFFF) corresponding to the character,, UTF-8 is the definition of the character corresponding to the number (not all the numbers have corresponding characters), after the end of a mapping between them

3/ F Munn

There's a typo in the first paragraph

Re: The man running

@ Munn, has changed, too careless, here under review, thank you for reminding

2/F Xiao Wan

What bloggers call Unicode is actually utf-16.

1/ F upfriend

I did not pay much attention to this problem before, because the general situation I let the front and back code unified, but Bo Master analysis is very good, praise!

