How to process the Chinese character string

This character is often found when processing crawled web pages. Try all kinds of transcoding, no result. For example, the original text of {code...} is: {code...}. How can this problem be solved? If you need more information, you can see it. �This character. Try all kinds of transcoding, no result.

For example:

Everyone tends to have preferences for their work.

The original text is:

Each person has a preference for using tools.

How can this problem be solved?

If you need information, you can check it out.

Http://apps.timwhitlock.info/unicode/inspect? S = % EF % BF % BD

Reply content:

When processing some crawled web pages, it is often found that�This character. Try all kinds of transcoding, no result.

For example:

Everyone tends to have preferences for their work.

The original text is:

Each person has a preference for using tools.

How can this problem be solved?

If you need information, you can check it out.

Http://apps.timwhitlock.info/unicode/inspect? S = % EF % BF % BD

When converting from an encoding to a Unicode encoding, if there is no corresponding character, the Unicode code "\ uffffd" will be obtained, that is�This character.
This is because your crawler does not recognize the encoding format (ASCII or GB2312, etc.) and compression format (Gzip, etc.) of the original webpage, and all of them become UTF-8 strings, this character indicates that the conversion has failed and the data has been lost. This character has no practical significance.

If it is PHP, this may be caused by substring. The solution is to install the mb_string module and use the mb _ * series functions.

Sometimes a part of a Chinese character is truncated to produce this symbol. For example, if a two-byte man is truncated to one byte, I don't know how to deal with it...
You should be split into two parts ..

public static function utf8Substr($str, $from, $len){    return preg_replace('#^(?:[\x00-\x7F]|[\xC0-\xFF][\x80-\xBF]+){0,'.$from.'}'.        '((?:[\x00-\x7F]|[\xC0-\xFF][\x80-\xBF]+){0,'.$len.'}).*#s',        '$1',$str);}

This problem is probably because the text is truncated during the line break. When you dig the text, check whether the line break has split the text into two halves.

It may be garbled.
You can try it in a browser.