Solution to iconv truncation in php

This article mainly introduces how to solve the iconv Chinese truncation problem in php, and analyzes in detail the causes and specific solutions of the Chinese truncation problem in the form of examples, which has some reference value, for more information about how to solve iconv truncation in php, see the following example. Share it with you for your reference. The specific analysis is as follows:

Today, I made a collection program. The principle is very simple. I used the curl method to obtain and analyze the html of the target page, and then extracted the required data using regular expressions and saved it in the database.

Because the opposite side of the page is GB2312 encoding, while the local use of UTF-8 encoding. Therefore, encoding conversion is required after collection.

The iconv method is used for encoding conversion.

Iconv-string conversion based on the required character encoding
String iconv (string $ in_charset, string $ out_charset, string $ str)

Encodes str from in_charset to out_charset.

The conversion method is simple. you can simply use the iconv method.

<? Php $ content = iconv ('gb2312', 'utf-8', $ content); // $ content is the collected content?>

After testing several pages, the data can be collected normally. However, in subsequent collection, the collection of several pages is incomplete.
Check the backend servers for regular expressions. After investigation, it is found that the content after iconv transcoding is much less than the collected content.
View the apache log and see the prompt: Notice: iconv (): Detected an illegal character in input string.

Refer to the manual and see the following description.

If you add a string // Transcoder after out_charset, the transliteration function is enabled. This means that when a character cannot be expressed by the target character set, it can be expressed in an approximate manner by one or more character similar to the character set.

If you add a string // IGNORE, characters that cannot be expressed in the target character set will be discarded silently. Otherwise, str is truncated from the first invalid character and leads to an E_NOTICE.

When iconv encounters unrecognized content, it truncates the content from the first unrecognizable character and generates an E_NOTICE. Therefore, the following content is discarded.

When the output character set is appended with // IGNORE, only unrecognized content is discarded, and subsequent content is not truncated or discarded.

After the program is modified, everything works normally.

<? Php $ content = iconv ('gb2312', 'utf-8 // IGNORE ', $ content); // $ content is the collected content?>

Tips: when using iconv, if you want to use UTF-8 encoding, use UTF-8 instead of UTF8, because UTF8 has some server issues.

I hope this article will help you with php programming.