Python + MongoDB crawler practice

Source: Internet
Author: User

Tool preparation and crawler building Scrapy (python-written crawler framework) in the previous Scrapy crawler entry, there is the installation and basic use of Scrapy, it is characteristic that each different page is customized with a different Spider, use the [java] scrapy crawl spidername-o file-t json method to run the crawler program and save it to the target file in json format (of course, the following items-o-t can be omitted ). In addition, there are few python code and the implementation is very convenient. You can easily get started with python syntax. By the way, I would like to explain my summary of the python syntax from the perspective of Java developers and python beginners: the end of the sentence does not contain semicolons (if), while, for without braces () and does not need to be declared, three main data results are simpler than the js Declaration: dictionary, list, And tuples with no good IDE. You should declare it by yourself: the package import should be self-declared, and the code should be short for exceptions, it gives me a strong sense of emptiness. MongoDB (with pymongo and mongo-java-driver) has long been trying to practice MongoDB. I have read the MongoDB authoritative guide before, but I have never had a chance to practice it, A loss of money can drive my demand. Because MongoDB stores objects in the form of BSON, I think it is suitable to store Scrapy json crawling results for interaction by pymongo. The crawled data will be handed over to solr to build a search service later, so we have installed the java driver to check the read results. Simple java-driver operations pymongo's simple operation data try to use data in the CSDN blog to crawl the Blog content data in the CSDN blog. First, crawl the names and URLs of some bloggers on the popular article page: [html] {'username': 'xxx', 'url': 'http: // blog.csdn.net/xxx/article/list/n'} crawls all the article url links of the blogger, crawls the content of each url link, and saves it to MongoDB: [html] 'user': 'xxx ', 'title': 'xxxxx', 'tag': ['XX', 'XX', 'xx'], 'content ': 'xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'} Some technologies are demand-driven to try and learn. Some problems can be discovered only when they have a certain degree of execution. What is MongoDB's read performance? How can he display automatic parts? How can mapreduce of built-in js version be used? How to Make Scrapy a service? Various practices of the python language? In the future, data will be designed for papers, personal homepages, And the CSDN blog data here will only pave the way. However, I am still looking forward to the entire learning and exploration process, haha.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.