Python3 provides an example of html webpage function that captures javascript dynamically generated, python3javascript

Python3 provides an example of html webpage function that captures javascript dynamically generated, python3javascript

This example describes how Python3 can capture html web pages dynamically generated by javascript. We will share this with you for your reference. The details are as follows:

Using urllib to capture a webpage can only read the static source file of the webpage, but cannot capture the content generated by javascript.

The reason is that urllib is instantaneous capture and does not wait for the loading delay of javascript. Therefore, the content generated by javascript in the page cannot be read by urllib.

Is there no way to read the content generated by javascript? None!

Here we will introduce a python Library: selenium. The version used in this article is 2.44.0.

First install:

pip install -U selenium

The following uses three examples to describe its usage:

[Example 0]

Open a Firefox browser
Load the page of the given url address

from selenium import webdriverbrowser = webdriver.Firefox()browser.get('http://www.baidu.com/')

[Example 1]

Open a Firefox browser
Go to Baidu Homepage
Search "seleniumhq"
Close your browser

From selenium import webdriverfrom selenium. webdriver. common. keys import Keysbrowser = webdriver. firefox () browser. get ('HTTP: // www.baidu.com ') assert 'Baidu' in browser. titleelem = browser. find_element_by_name ('P') # Find the search boxelem. send_keys ('seleniumhq '+ Keys. RETURN) # simulate the browser button. quit ()

[Example 2]

Selenium WebDriver is often used for testing network programs. The following is an example of using the Python standard library unittest:

Import unittestclass BaiduTestCase (unittest. testCase): def setUp (self): self. browser = webdriver. firefox () self. addCleanup (self. browser. quit) def testPageTitle (self): self. browser. get ('HTTP: // www.baidu.com ') self. assertIn ('Baidu ', self. browser. title) if _ name _ = '_ main _': unittest. main (verbosity = 2)