Python Crawl Watercress Group 700+ topic add back to La Python open file with a variable name

Source: Internet
Author: User

Requirements: Crawl all the topics of the Watercress Group (topic title, content, author, release time), and reply (best reply, normal reply, Reply _ reply, page back, 0 replies.)

Solution: 1. First crawl under the group, all the topic links, through the positioning of nextpage page to get the total 700+ article topic;

2. Visit the 700+ link, in the +start=0 of the page, get four parts related to the topic (title, content, author, release time), and the best reply, reply;

3. On the basis of 2, to determine whether there is a reply, if there is a reply to further determine whether there is a reply to the page, reply to pages through nextpage to obtain start=100, start=200;

4. Go to the next crawl function, and continue to write the crawled reply to the file in 2;

Solution Ideas:

before: started to build 2 files, Article.txt used to store all topics related content (700+ topic, author information);

At the same time, the establishment of a title named reply file;

After : The unified establishment of the topic title of the article, first write the article related content, and then continue to write back, so easy to read;

The pits encountered:

1. To get the text under a div under direct text,div.span under Text,div.h:

-There are 2 workarounds:

A. Through the XPath//text, which means to get all the text files under the div;

B. With CSS stitching, commas can be separated:

2. Consolidates the method of passing parameters between different functions via meta:

3. Python open file with a variable name

f = open ('%s.txt '% title_end, ' a ')

A: Continue writing

4. Remove the spaces in Str, line breaks, and other symbols

# Remove the whitespace, \t,\n and \ r characters around X.
X1 = X.strip (' \t\n\r ')


5. Strip remove the \ r in the data, '. Join reverses the list back to the string;

# first to remove the \ r in the article, some separate ' \ R ' becomes an empty list element: ", and then use if to judge the next good
Artical_end = []
For x in article:
X1 = X.replace (' \ R ', ')
If x1! = ":
Artical_end.append (x1)
# Convert Artical_end list to string
AR = '. Join (Artical_end)

Python Crawl Watercress Group 700+ topic add back to La Python open file with a variable name

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.