(Python parse JS) selenium combined with Phantomjs crawl JS generated page

, some pages are dynamically created at load time HTML content, as long as the JS code fully executed after the final results will be displayed. If you crawl a page in a traditional way, you can only get the content on the page before the JS code executes.

There are two ways to solve this problem:

  1. Crawl data directly from JS code (execute JS code, parse JS variable).
  2. Run JS with third party library, crawl the final HTML page after running.

using selenium in Python to execute JS

Selenium is a powerful network data acquisition tool, which was originally developed for Web site automation testing. Selenium allows browsers to automatically load pages, get the data they need, even screen screenshots, or determine whether certain actions on the site occur.

Selenium itself without a browser, it needs to be used in conjunction with Third-party browsers. Here you use the PHANTOMJS tool instead of the real browser.

PHANTOMJS is a WebKit-based server-side JavaScript API. It fully supports the web without the need for browser support, its fast, native support for a variety of Web standards: DOM processing, CSS selectors, JSON, Canvas, and SVG. PHANTOMJS can be used for page automation, network monitoring, web screen screenshots, and no interface testing.

By combining selenium with PHANTOMJS, you can run a very powerful reptile that can handle cookie,js,header and anything you need to do. Installation:

Selenium has a Python library that can be installed with Pip and so on; PHANTOMJS is a full-featured "headless" browser, not a Python library, so it does not need to be installed like other libraries in Python, nor can it be installed with PIP.

sudo pip install selenium
http://npm.taobao.org/dist/phantomjs/ 
#下载安装包 (sudo apt-get install PHANTOMJS installed is not up to date, discovery cannot be used)
phantomjs-2.1.1-linux-i686.tar.bz2
tar-jxvf phantomjs-2.1.1-linux-i686.tar.bz2

use:

From selenium import webdriver
driver = webdriver. Phantomjs (executable_path= '/opt/phantomjs-2.1.1-linux-i686/bin/phantomjs ')
#executable_ path for your phantomjs executable paths
driver.get ("http://news.sohu.com/scroll/")

#或得js变量的值
r = Driver.execute_ Script ("Return Newsjason")
print R

#selenium在webdriver的DOM中使用选择器来查找元素, the name is direct when the by object can be used by the selection strategy: Id,class_ Name,css_selector,link_text,name,tag_name,tag_name,xpath and so on
print driver.find_element_by_tag_name ("div"). Text
print driver.find_element_by_csss_selector ("#content"). Text
print driver.find_element_by_id (" Content "). Text