This paper makes a deep analysis of the common modules of Python crawler, and explains them in detail with examples. Share for everyone to use for reference. The specific analysis is as follows:
Creepy module
A Taiwan great God developed, the function is simple, can automatically crawl a website of all content, of course, you can also set which URLs need to catch.
Address: Https://pypi.python.org/pypi/creepy
Function Interface:
Set_content_type_filter:
Sets the content-type of the grab (the ContentType in the header). including text/html
Add_url_filter:
Filter URL, incoming can be a regular expression
Set_follow_mode:
Set recursive mode, F_any: All links on this page will crawl. F_same_domain and F_same_host are similar. That is, the same domain name will crawl. F_same_path: Crawl of the same path. For example, bag.vancl.com/l1/d3/1.jpg path is l1/d3/1.jpg, then the path is l1/d3/* and will crawl. Here you can add your own recursive pattern as needed
Set_concurrency_level:
Set maximum number of threads
Process_document:
Generally need to rewrite, processing Web content, extract the content of their own needs.
Selenium
Visual interface, crawl Automation, API use is very simple, exactly like the operation of the browser itself.
Official website: http://www.seleniumhq.org/
Python official website
Http://pypi.python.org/pypi/selenium
Webdriver API (very useful, recommended to understand more)
http://www.seleniumhq.org/docs/03_webdriver.jsp
Here is an example of a crawling site for visitors:
From selenium import webdriver from
selenium.webdriver.common.keys import keys
import time
browser = Webdriver. Firefox ()
browser.get (' Http://bag.vancl.com/28145-28167-a18568_18571-b1-n3-s1.html#ref=hp-hp-hot-8_1_1-v: n ')
elem = Browser.find_element_by_name (' Ch_bag-3-page-next ') # Find the search box
time.sleep (1)
print Elem.get_attribute ("href")
Elem.click ()
time.sleep (1)
elem = Browser.find_element_by_name (' Ch_ Bag-3-page-next ') # Find the search box
print elem.get_attribute ("href")
Elem.click ()
I hope this article will help you with your Python programming.