"Flying Valley Six" crawler Project 2

Probably know some ideas, try to push the net crawl.

The first URL is in the format: Http://www.neitui.me/?name=neitui&handle=lists&keyword={key_word}&page={cur_page}

After the URL question mark followed by the parameter, the equals sign has the value is the valid parameter, the non-value can omit;& is the connection symbol, is used for the connection parameter.

Through the XPath extract page, I know today, originally in Google browser with F12 display source code, you can press ctrl+f after the XPath expression to find the target, so after writing the XPath expression can be directly verified on the page, it is easy to determine whether to simplify the expression.

The first is to obtain the total number of posts in the specified position, through the analysis source know the source of the total number of pages this article. Analysis page

The upper part of the page will give the total number of positions, a fixed 28 positions per page, which seems to be able to get the number of pages by (total positions/28) +.

However, after trying several pages, I found that when the total number of posts exceeds 1000, the display looks like this:

This will not get the number of pages by division.

See the page number section again

You can see that there is no backward icon on the page if there is no next page. The source code can be analyzed by

div[@class = "T_pagelink"]//a[@class = "Next"]

(analysis found can be reduced to//a[@class = "Next"])

To find the backward icon.

This will require each page to be crawled after the analysis of whether the next page, if any, continue to crawl.

"Flying Valley Six" crawler Project 2