I need to collect the content pages of a website. Collect the Link first, and then obtain the content page through the link. However, if there are too many links, the retrieval times out and requires a small amount of retrieval by category. There are more than 300 categories. Setting categories manually is also slow. How can I automatically retrieve the data I want without collecting the content pages of a website. Collect the Link first, and then obtain the content page through the link. However, if there are too many links, the retrieval times out and requires a small amount of retrieval by category. There are more than 300 categories. Setting categories manually is also slow. How can I automatically obtain the desired data without timeout? Please refer to your ideas. Thank you!
Sorry, I didn't make it clear. The timeout value is 504 Gateway timeout.
Setting set_time_limit is useless.
The problem has been disabled. cause:
Reply content:
I need to collect the content pages of a website. Collect the Link first, and then obtain the content page through the link. However, if there are too many links, the retrieval times out and requires a small amount of retrieval by category. There are more than 300 categories. Setting categories manually is also slow. How can I automatically obtain the desired data without timeout? Please refer to your ideas. Thank you!
Sorry, I didn't make it clear. The timeout value is 504 Gateway timeout.
Setting set_time_limit is useless.
Why can I try scrapy with php?
504 gateway times out. Your Web server crashes. You can run the php script directly by running the command line. Is Apache or Nginx used on your server?
Well, timeout?set_time_limit(0);You can keep the script running ~
You can modify the max_execution_time value of php. ini, or use the set_time_limit function as described above.
The collection is executed on the command line and does not time out :)