Target site:Http://news.xinhuanet.com/photo/2015-01/29/c_127433852.htm
Get content:Body picture and text description;
I implement the method:
Because the paging address is sequential, I do a loop on the destination address.
http://news.xinhuanet.com/photo/2015-01/29/c_127433852_2. htm
get_headersgets the return after access is 200, which is the words to crawl the content
1. After curl gets the current address HTML, intercepts the required content
2,_2 This is the variable, since the increase to continue to determine whether the next page can be accessed, can be accessed to continue to collect
The above is possible to achieve capture, but it seems that the speed is very slow, and each time it is curl to crawl to the server seems to have pressure
question: How to improve, how do you crawl with PHP, ask for ideas and direction
Reply content:
target site:http://news.xinhuanet.com/photo/2015-01/29/c_127433852.htm
get content: body picture and text description;
I implement the method:
Because the paging address is sequential, I do a loop on the destination address.
http://news.xinhuanet.com/photo/2015-01/29/c_127433852_2. htm
get_headersgets the return after access is 200, which is the words to crawl the content
1. After curl gets the current address HTML, intercepts the required content
2,_2 This is the variable, since the increase to continue to determine whether the next page can be accessed, can be accessed to continue to collect
The above is possible to achieve capture, but it seems that the speed is very slow, and each time it is curl to crawl to the server seems to have pressure
question: How to improve, how do you crawl with PHP, ask for ideas and direction
The following is not a page, direct access to the inside of the link to the array grab it
- Directly crawl, when the return of the results of the time to determine whether the status code is 200
- Curl concurrency