SELENIUM2+PHANTOMJS Introductory Example

This is my study of reptiles more in-depth step, most of the Web crawl with URLLIB2 can be done, but when it comes to JavaScript, Urlopen is completely stupid, so have to use the simulation browser, there are many methods, here I use is selenium2+ PHANTOMJS, the reason is:

SELENIUM2 supported all the major browsers and PHANTOMJS browsers, I started to use chrome, but found that I needed to install a chrome driver, so I deprecated, chose Phantomjs, and the name sounded more foreign flavor.

On the internet to check a lot of information, found that online selenium2+phantomjs use of Chinese resources is very deficient, have to read obscure official documents, so here to write down the current implementation of the operation, plus some of my personal problems and corresponding solutions.

Background knowledge:

PHANTOMJS is a webkit-based browser with no interface, so it's more efficient to run than a full browser.

Selenium is a tool for testing Web applications, currently 2.42. Version 1, and version 1 is the difference between the 2.0+ webdrive integration.

Selenium2 supported Python versions:2.7, 3.2, 3.3 and 3.4

Additional installation of Selenium server is required if remote operation is required

Installation:

First installed selenium2, which way can be installed, I usually directly download the compressed package, and then use the Python setup.py install command to install, Selenium 2.42.1: https://pypi.python.org/pypi/ selenium/2.42.1

Then download Phantomjs,https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-1.9.7-windows.zip, unzip to see a Phantomjs.exe file

Example 1:

#Coding=utf-8 fromSeleniumImportWebdriverdriver= Webdriver. PHANTOMJS (executable_path='C:\Users\Gentlyguitar\Desktop\phantomjs-1.9.7-windows\phantomjs.exe') Driver.get ("http://duckduckgo.com/") driver.find_element_by_id ('Search_form_input_homepage'). Send_keys ("Nirvana") driver.find_element_by_id ("Search_button_homepage"). Click ()Printdriver.current_urldriver.quit ()

The Executable_path is just the path of the Phantomjs.exe, the result of the operation:

Https://duckduckgo.com/?q=Nirvana

Walk through of the example:

It is worth mentioning that:

The Get method waits until the page is fully loaded before proceeding with the program

But for Ajax: It's worth noting that if your page uses a lot of Ajax in load then webdriver if the know when it had compl Etely Loaded

Send_keys is the fill input

Example 2:

#Coding=utf-8 fromSeleniumImportWebdriver fromSelenium.webdriver.common.keysImportKeys fromSelenium.webdriver.support.uiImportwebdriverwait fromSelenium.webdriverImportActionchainsImport TimeImportSysdriver= Webdriver. PHANTOMJS (executable_path='C:\Users\Gentlyguitar\Desktop\phantomjs-1.9.7-windows\phantomjs.exe') Driver.get ("http://www.zhihu.com/#signin")#driver.find_element_by_name (' email '). Send_keys (' your email ')Driver.find_element_by_xpath ('//input[@name = "password"]'). Send_keys ('Your password')#Driver.find_element_by_xpath ('//input[@name = "password"]). Send_keys (Keys.return)Time.sleep (2) Driver.get_screenshot_as_file ('Show.png')#Driver.find_element_by_xpath ('//button[@class = "Sign-button"]). Click ()Driver.find_element_by_xpath ('//form[@class = "Zu-side-login-box"]'). Submit ()Try: Dr=webdriverwait (driver,5) Dr.until (LambdaThe_driver:the_driver.find_element_by_xpath ('//a[@class = "Zu-top-nav-userinfo"]'). is_displayed ())except:    Print 'Logon Failure'sys.exit (0) Driver.get_screenshot_as_file ('Show.png')#user=driver.find_element_by_class_name (' Zu-top-nav-userinfo ')#Webdriver. Actionchains (Driver). Move_to_element (user). Perform () #移动鼠标到我的用户名Loadmore=driver.find_element_by_xpath ('//a[@id = "Zh-load-more"]') Actions=actionchains (Driver) actions.move_to_element (Loadmore) Actions.click (Loadmore) actions.perform () Time.sleep ( 2) Driver.get_screenshot_as_file ('Show.png')PrintDriver.current_urlPrintdriver.page_sourcedriver.quit ()

This program is completed, login to know, and then can automatically click on the page below the "more" to load more content

Walk through of the example:

From Selenium.webdriver.common.keys import Keys,keys This class is the key on the keyboard, the text of the Send_keys (Keys.return) is to press a carriage return

From Selenium.webdriver.support.ui import webdriverwait is for a later wait operation

From Selenium.webdriver import Actionchains is the class that imports an action, the wording of this sentence, I looked for a long time

Find_element recommended the use of XPath method, because: forcing lattice, and really very convenient

Syntax for XPath expressions tutorial: http://www.ruanyifeng.com/blog/2009/07/xpath_path_expressions.html

It is worth noting that you should avoid selecting the value with a space attribute, such as class = "Country name", otherwise it will be an error, probably compound class or something wrong

The correct way to check the user's password is to take a screenshot after filling it in.

If you want to get a screenshot, this is the line:

driver.get_screenshot_as_file ('show.png')

However, the screenshot here is not with the scroll bar, is to give you the entire page photo down

Try :    Dr=webdriverwait (driver,5)    dr.until (Lambda the_driver:the_driver.find_ Element_by_xpath ('//a[@class = "Zu-top-nav-userinfo"]). is_displayed ())  Except:    print' Login failed '    sys.exit (0) 

is used to check if an element is loaded to see if the login is successful, I think it is possible to use a black box. 5 of the Explanations: 1 page changes are scanned every 500 milliseconds in 5 seconds until the specified element

For a form submission, you can select the login button and then use the click Method, or you can select the form and then use the Submit method, which can handle the absence of a login button, so it is recommended to use the Submit ()

For a single click, you can either use the click () or use a series of actions, as in the text:

Loadmore=driver.find_element_by_xpath ('//a[@id = "Zh-load-more"]'= actionchains (Driver) actions.move_to_element (Loadmore) Actions.click (Loadmore) actions.perform ()

These 5 sentences are actually equivalent to a sentence, find element and click, but the scope of action is more extensive, for example, in this case, to click on a tag object, I do not know why directly with the click does not work

Print Driver.current_url Print Driver.page_source

Print two properties of a Web page: URL and source

Reference documents:

Http://www.realpython.com/blog/python/headless-selenium-testing-with-python-and-phantomjs/#.U5FXUvmSziE

Http://selenium-python.readthedocs.org/getting-started.html

Http://www.ruanyifeng.com/blog/2009/07/xpath_path_expressions.html

Http://www.cnblogs.com/paisen/p/3310067.html