Continuous Paging collection problem

Source: Internet
Author: User
Target site:Http://news.xinhuanet.com/photo/2015-01/29/c_127433852.htm
Get content:Body picture and text description;
I implement the method:
Because the paging address is sequential, I do a loop on the destination address.

http://news.xinhuanet.com/photo/2015-01/29/c_127433852_2. htm

get_headersgets the return after access is 200, which is the words to crawl the content
1. After curl gets the current address HTML, intercepts the required content
2,_2 This is the variable, since the increase to continue to determine whether the next page can be accessed, can be accessed to continue to collect

The above is possible to achieve capture, but it seems that the speed is very slow, and each time it is curl to crawl to the server seems to have pressure

question: How to improve, how do you crawl with PHP, ask for ideas and direction

Reply content:

target site:http://news.xinhuanet.com/photo/2015-01/29/c_127433852.htm
get content: body picture and text description;
I implement the method:
Because the paging address is sequential, I do a loop on the destination address.

http://news.xinhuanet.com/photo/2015-01/29/c_127433852_2. htm

get_headersgets the return after access is 200, which is the words to crawl the content
1. After curl gets the current address HTML, intercepts the required content
2,_2 This is the variable, since the increase to continue to determine whether the next page can be accessed, can be accessed to continue to collect

The above is possible to achieve capture, but it seems that the speed is very slow, and each time it is curl to crawl to the server seems to have pressure

question: How to improve, how do you crawl with PHP, ask for ideas and direction

The following is not a page, direct access to the inside of the link to the array grab it

    1. Directly crawl, when the return of the results of the time to determine whether the status code is 200
    2. Curl concurrency
  • Contact Us

    The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

    If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.