Php collection program

Source: Internet
Author: User
After the php collection program completes the collection for about 10 minutes, fwrite (...) appears (......) request, warning, and packet loss. some Web pages may fail to be crawled ..

No problem in the first 10 minutes. after 10 minutes, the frequency of sending requests to fwrite (...) will gradually increase.

Is the request rejected by the server?

Error message:

Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6565_6.html
The next group will be collected soon
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_2.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_3.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_4.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_5.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_6.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_7.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_8.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_9.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6559_10.html
The next group will be collected soon
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6549.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6549_2.html

Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6549_3.html
The next group will be collected soon

Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200
Request the initiator of the link http://www.mmkao.com/beautyleg/201511/6537.html

Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 200

A bit confused, the implementation principle of the code is still very simple, but it is difficult to debug network requests. if you have experience in this field, please share it.


Is there a lot of code? The system prompts that the length cannot exceed 10000 characters.
Code link here: http://blog.csdn.net/free_program_1314/article/details/41798199


Reply to discussion (solution)

$ This-> conn = fsockopen ($ this-> ip, $ this-> port, $ errno, $ errstr, $ this-> timeout) at RoW 100 );
Then, the system checks whether the connection is successful in row 3. if the connection fails, the system returns false.
However, when you call $ this-> connect () in line 62, the system does not check whether false is returned.
That is, the program continues to run no matter whether the connection is successful or not. As a result, it is natural to spread the hidden danger out.
You must determine whether the $ this-> conn is a resource before fwrite ($ this-> conn,...) in line 200.

If $ this-> connect () fails, it is meaningless to go to the policy processing program and follow the normal process.

$ This-> conn = fsockopen ($ this-> ip, $ this-> port, $ errno, $ errstr, $ this-> timeout) at RoW 100 );
Then, the system checks whether the connection is successful in row 3. if the connection fails, the system returns false.
However, when you call $ this-> connect () in line 62, the system does not check whether false is returned.
That is, the program continues to run no matter whether the connection is successful or not. As a result, it is natural to spread the hidden danger out.
You must determine whether the $ this-> conn is a resource before fwrite ($ this-> conn,...) in line 200.

If $ this-> connect () fails, it is meaningless to go to the policy processing program and follow the normal process.



If ($ this-> conn) {if ($ len! = Fwrite ($ this-> conn, $ this-> consumer _line. $ this-> header. $ this-> post_content, $ len) {$ this-> status = "sending request failed"; die ($ this-> status );}} else {die ($ this-> status );}

// The error occurs when the fwrite (...) fails to send the request...
Notice: fwrite (): in E: \ CodeEdit \ php \ http \ 001.php on line 213
Send request failed

You have not determined whether the $ this-> conn is valid.

You have not determined whether the $ this-> conn is valid.



A little improved. the link is okay. the error prompt is:

Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_9.html
No next page exists because: An error occurred while matching the image paging group.

However, the getRollLink function that matches the next page does not have any problems. I have tested it over and over again, even if it reads data from the locally saved webpage, and before this problem occurs, I have collected a lot of Web pages, and there is no problem.

Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_5.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_6.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_7.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_8.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201412/7066_9.html
No next page exists because: An error occurred while matching the image paging group.

After the improvements are posted here: http://blog.csdn.net/free_program_1314/article/details/41805697

An example is provided to illustrate this problem. for example, after the ninth page is collected, the matching fails when the tenth page is collected, and the link to the tenth page is provided by the ninth page. Open the ninth collected file and read and match the link on the next page. This is the case multiple times. every time I try to read from the file and match the link on the next page, it is okay.


That is to say, match the link on the next page. if there is a problem, this prompt should appear at the beginning of the collection... The problem is that this problem does not occur from time to time.

It is normal to appear occasionally, even if it is accessed through a browser, sometimes the page cannot be opened (refresh it) is not normal?
You only need to check the connection validity at any time and handle the failure. For example, wait a few seconds to connect again.

It is normal to appear occasionally, even if it is accessed through a browser, sometimes the page cannot be opened (refresh it) is not normal?
You only need to check the connection validity at any time and handle the failure. For example, wait a few seconds to connect again.



I understand

It is normal to appear occasionally, even if it is accessed through a browser, sometimes the page cannot be opened (refresh it) is not normal?
You only need to check the connection validity at any time and handle the failure. For example, wait a few seconds to connect again.



This problem is solved by extracting the next page link from the saved previous page when the matching fails:

Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6906_13.html
No next page exists because: An error occurred while matching the image paging group. try to read the matching mode from the file.
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6906_14.html
The next group will be collected soon
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6892.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6892_2.html
Request the initiator of the link http://www.mmkao.com/beautyleg/201411/6892_3.html


After 10 minutes of collection, the problem of request failure still occurs, and fwrite () is another old problem. it should be solved by resetting the connection.

Looks like NB.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.