After the php collection program completes the collection for about 10 minutes, fwrite (...) appears (......) request, warning, and packet loss. some Web pages may fail to be crawled ..
No problem in the first 10 minutes. after 10 minutes, the frequency of sending requests to fwrite (...) will gradually increase.
Is the request rejected by the server?
Error message:
Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6565_6.html
The next group will be collected soon
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_2.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_3.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_4.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_5.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_6.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_7.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_8.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_9.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_10.html
The next group will be collected soon
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6549.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6549_2.html
Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6549_3.html
The next group will be collected soon
Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6537.html
Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
A bit confused, the implementation principle of the code is still very simple, but it is difficult to debug network requests. if you have experience in this field, please share it.
Is there a lot of code? The system prompts that the length cannot exceed 10000 characters.
Code link here: http://blog.csdn.net/free_program_1314/article/details/41798199
Reply to discussion (solution)
$ This-> conn = fsockopen ($ this-> ip, $ this-> port, $ errno, $ errstr, $ this-> timeout) at RoW 100 );
Then, the system checks whether the connection is successful in row 3. if the connection fails, the system returns false.
However, when you call $ this-> connect () in line 62, the system does not check whether false is returned.
That is, the program continues to run no matter whether the connection is successful or not. As a result, it is natural to spread the hidden danger out.
You must determine whether the $ this-> conn is a resource before fwrite ($ this-> conn,...) in line 200.
If $ this-> connect () fails, it is meaningless to go to the policy processing program and follow the normal process.
$ This-> conn = fsockopen ($ this-> ip, $ this-> port, $ errno, $ errstr, $ this-> timeout) at RoW 100 );
Then, the system checks whether the connection is successful in row 3. if the connection fails, the system returns false.
However, when you call $ this-> connect () in line 62, the system does not check whether false is returned.
That is, the program continues to run no matter whether the connection is successful or not. As a result, it is natural to spread the hidden danger out.
You must determine whether the $ this-> conn is a resource before fwrite ($ this-> conn,...) in line 200.
If $ this-> connect () fails, it is meaningless to go to the policy processing program and follow the normal process.
If ($ this-> conn) {if ($ len! = Fwrite ($ this-> conn, $ this-> consumer _line. $ this-> header. $ this-> post_content, $ len) {$ this-> status = "sending request failed"; die ($ this-> status );}} else {die ($ this-> status );}
// The error occurs when the fwrite (...) fails to send the request...
Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 213
Send request failed
You have not determined whether the $ this-> conn is valid.
You have not determined whether the $ this-> conn is valid.
A little improved. the link is okay. the error prompt is:
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_9.html
No next page exists because: An error occurred while matching the image paging group.
However, the getRollLink function that matches the next page does not have any problems. I have tested it over and over again, even if it reads data from the locally saved webpage, and before this problem occurs, I have collected a lot of Web pages, and there is no problem.
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_5.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_6.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_7.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_8.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_9.html
No next page exists because: An error occurred while matching the image paging group.
After the improvements are posted here: http://blog.csdn.net/free_program_1314/article/details/41805697
An example is provided to illustrate this problem. for example, after the ninth page is collected, the matching fails when the tenth page is collected, and the link to the tenth page is provided by the ninth page. Open the ninth collected file and read and match the link on the next page. This is the case multiple times. every time I try to read from the file and match the link on the next page, it is okay.
That is to say, match the link on the next page. if there is a problem, this prompt should appear at the beginning of the collection... The problem is that this problem does not occur from time to time.
It is normal to appear occasionally, even if it is accessed through a browser, sometimes the page cannot be opened (refresh it) is not normal?
You only need to check the connection validity at any time and handle the failure. For example, wait a few seconds to connect again.
It is normal to appear occasionally, even if it is accessed through a browser, sometimes the page cannot be opened (refresh it) is not normal?
You only need to check the connection validity at any time and handle the failure. For example, wait a few seconds to connect again.
I understand
It is normal to appear occasionally, even if it is accessed through a browser, sometimes the page cannot be opened (refresh it) is not normal?
You only need to check the connection validity at any time and handle the failure. For example, wait a few seconds to connect again.
This problem is solved by extracting the next page link from the saved previous page when the matching fails:
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6906_13.html
No next page exists because: An error occurred while matching the image paging group. try to read the matching mode from the file.
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6906_14.html
The next group will be collected soon
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6892.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6892_2.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6892_3.html
After 10 minutes of collection, the problem of request failure still occurs, and fwrite () is another old problem. it should be solved by resetting the connection.
Looks like NB.