Data collection-PHP collection timeout

Source: Internet
Author: User
I need to collect the content pages of a website. Collect the Link first, and then obtain the content page through the link. However, if there are too many links, the retrieval times out and requires a small amount of retrieval by category. There are more than 300 categories. Setting categories manually is also slow. How can I automatically retrieve the data I want without collecting the content pages of a website. Collect the Link first, and then obtain the content page through the link. However, if there are too many links, the retrieval times out and requires a small amount of retrieval by category. There are more than 300 categories. Setting categories manually is also slow. How can I automatically obtain the desired data without timeout? Please refer to your ideas. Thank you!

Sorry, I didn't make it clear. The timeout value is 504 Gateway timeout.
Setting set_time_limit is useless.

The problem has been disabled. cause:

Reply content:

I need to collect the content pages of a website. Collect the Link first, and then obtain the content page through the link. However, if there are too many links, the retrieval times out and requires a small amount of retrieval by category. There are more than 300 categories. Setting categories manually is also slow. How can I automatically obtain the desired data without timeout? Please refer to your ideas. Thank you!

Sorry, I didn't make it clear. The timeout value is 504 Gateway timeout.
Setting set_time_limit is useless.

Why can I try scrapy with php?

504 gateway times out. Your Web server crashes. You can run the php script directly by running the command line. Is Apache or Nginx used on your server?

Well, timeout?set_time_limit(0);You can keep the script running ~

You can modify the max_execution_time value of php. ini, or use the set_time_limit function as described above.

The collection is executed on the command line and does not time out :)

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.