Python3 crawling JavaScript-generated HTML pages

Crawl Web pages with Urllib and so on, can only read static source files of Web pages, and can't catch content generated by JavaScript.

The reason is that because urllib is instantaneous, it does not wait for JavaScript loading delay, so the content generated by JavaScript in the page is not read by Urllib.

Is there really no way to read the content generated by JavaScript? Not too!

This article introduces a python library: Selenium, the latest version is 2.44.0

Install First:

Pip Install-u Selenium

Here are three examples of how to use it:

"Example 0"

• Open a Firefox browser
• Page loaded with the given URL address

1  from Import Webdriver 2 3 browser = webdriver. Firefox ()4 browser.get ('http://www.baidu.com/')

"Example 1"

• Open a Firefox browser
• Download Baidu homepage
• Search "SELENIUMHQ"
• Close browser

1  fromSeleniumImportWebdriver2  fromSelenium.webdriver.common.keysImportKeys3 4Browser =Webdriver. Firefox ()5 6Browser.get ('http://www.baidu.com')7 assert 'Baidu' inchBrowser.title8 9Elem = Browser.find_element_by_name ('P')#Find the search boxTenElem.send_keys ('SELENIUMHQ'+keys.return) # analog keys  One  ABrowser.quit ()

"Example 2"

Selenium Webdriver is often used for network program testing. Here is an example of using the Python standard library unittest:

1 ImportUnitTest2 3 classbaidutestcase (unittest. TestCase):4 5     defsetUp (self):6Self.browser =Webdriver. Firefox ()7 Self.addcleanup (self.browser.quit)8 9     defTestpagetitle (self):TenSelf.browser.get ('http://www.baidu.com') OneSelf.assertin ('Baidu', Self.browser.title) A  - if __name__=='__main__': -Unittest.main (verbosity=2)

Python3 crawling JavaScript-generated HTML pages