A common modular analysis _python python crawler

Source: Internet
Author: User

This paper makes a deep analysis of the common modules of Python crawler, and explains them in detail with examples. Share for everyone to use for reference. The specific analysis is as follows:

Creepy module

A Taiwan great God developed, the function is simple, can automatically crawl a website of all content, of course, you can also set which URLs need to catch.

Address: Https://pypi.python.org/pypi/creepy

Function Interface:

Set_content_type_filter:
Sets the content-type of the grab (the ContentType in the header). including text/html

Add_url_filter:
Filter URL, incoming can be a regular expression

Set_follow_mode:
Set recursive mode, F_any: All links on this page will crawl. F_same_domain and F_same_host are similar. That is, the same domain name will crawl. F_same_path: Crawl of the same path. For example, bag.vancl.com/l1/d3/1.jpg path is l1/d3/1.jpg, then the path is l1/d3/* and will crawl. Here you can add your own recursive pattern as needed

Set_concurrency_level:
Set maximum number of threads

Process_document:
Generally need to rewrite, processing Web content, extract the content of their own needs.

Selenium
Visual interface, crawl Automation, API use is very simple, exactly like the operation of the browser itself.

Official website: http://www.seleniumhq.org/
Python official website
Http://pypi.python.org/pypi/selenium
Webdriver API (very useful, recommended to understand more)
http://www.seleniumhq.org/docs/03_webdriver.jsp

Here is an example of a crawling site for visitors:

From selenium import webdriver from
selenium.webdriver.common.keys import keys
import time

browser = Webdriver. Firefox ()
browser.get (' Http://bag.vancl.com/28145-28167-a18568_18571-b1-n3-s1.html#ref=hp-hp-hot-8_1_1-v: n ')
elem = Browser.find_element_by_name (' Ch_bag-3-page-next ') # Find the search box
time.sleep (1)
print Elem.get_attribute ("href")
Elem.click ()

time.sleep (1)
elem = Browser.find_element_by_name (' Ch_ Bag-3-page-next ') # Find the search box
print elem.get_attribute ("href")
Elem.click ()

I hope this article will help you with your Python programming.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.