Python crawler Learning (9): Use of Selenium

1 Introduction and Installation

What is Selenium? In a word, automated testing tools. It supports a variety of browsers, including Chrome,safari,firefox and other mainstream interface browser, if you install a Selenium plug-in in these browsers, then you can easily implement the Web interface testing. In other words, call Selenium to support these browser drivers. Selenium support multiple language development, such as Java,c,ruby and so on, have Python? That's a must! Oh, this is really great news.

Selenium 2, aka Webdriver, its main new feature is the integration of Selenium 1.0 and Webdriver (Webdriver used to be Selenium's competitor). That is to say, Selenium 2 is a merger of Selenium and Webdriver two projects, Selenium 2 compatible Selenium, which supports both Selenium API and Webdriver API.

Install: If PIP is already installed, you can install it directly from:

Pip Install Selenium

Document: http://selenium-python.readthedocs.io/

2. Simple to use

2.1 Open Browser

# !/usr/bin/python  from Import webdriverbrowser=webdriver. Firefox () browser.get ("http://www.baidu.com")

Error:

Program execution error, browser does not open, then should not be the Firefox driver is not configured in the environment variable. Download the driver, and then configure the drive file path in the environment variable.

: https://github.com/mozilla/geckodriver/releases/

After the download is done, it is pressurized into the/usr/local/bin directory and then executed, and we see the browser pop-up directly and its page is Baidu.

2.2 Sending data

For the convenience of interaction, the following operation is done directly with Ipython. (If not, you can install it directly in Centos7: sudo yum install ipython).
After the installation is complete, you can enter Ipython startup Ipython directly into the command, and then you can enter the Python statement in it with the tab completion function. The bottom example is to open Firefox

　　

Execute the following words in turn:

 from Import Webdriverin [fromimport  Keys#  Open Browser in [3]: Browser =  Webdriver. Firefox ()#  where the Driver.get method will open the requested url,webdriver will wait until the page is fully loaded before returning,
# that is, the program will wait for all the contents of the page to finish loading, JS rendering is completed before continuing to execute.
Note: If there is a lot of Ajax to use here, the program may not know if it is fully loaded. in [4]: Browser.get ("http://www.baidu.com")

Then we through Firebug (in Firefox press F12), find out the name of the Baidu input box. The name of the Discovery input box is: WD

　　

Next, we enter the values in this input box via selenium:

# Find the page input box by element name In [5]: Elem = Browser.find_element_by_name ("wd")# Enter the contents of the query in the input box in [6]: Elem.send_keys ("selenium")

At this point we find that the query value has already appeared in the page:

　　

You can then send a down arrow to indicate that it is selected in the above options:

　　

Elem.send_keys (Keys.arrow_down)

Then we can enter a carriage return through selenium to query, the results can be viewed directly in the browser. (is not feeling very strong, through the statement Operation Browser)

In [7]: Elem.send_keys (Keys.return)

After that we can get the rendered page via Browser.page_source

3. Interaction

Part 2.2 of the preceding section already explains the use of interactions. This will be explained further here.

3.1 Acquisition of elements

When there are elements that are defined as follows:

<type= "text"  name= "passwd"  ID= "Passwd-id" />

We can navigate to this element in several ways:

# get this element by element ID elements = driver.find_element_by_id ("passwd-id")#  get elements by name element element = Driver.find_element_by_name ("passwd") # get element by XPath = Driver.find_element_by_xpath ("//input[@id = ' Passwd-id ') ")

# get elements by tag name
element = Driver.find_element_by_tag_name
。。 Other

Note: If the corresponding element is not found, an exception is thrown NoSuchElementException . The retrieved elements are represented by a class, which means that if you use the IDE, you are automatically prompted by many methods that are not available for each one.

XPath is a language that looks for information in an XML document. XPath can be used to iterate over elements and attributes in an XML document, or in HTML, with little knowledge of XPath, and if you want to learn, click on the address below:

　　XPath Tutorial

　　3.2 Selection

We have successfully obtained and entered into the text box, and now we are going to manipulate the following other elements. Like we're going to go down the dropdown box, we can do this:

#gets the drop-down elementelement = Driver.find_element_by_xpath ("//select[@name = ' name ']")#get all the options in the drop-downAll_options = Element.find_elements_by_tag_name ("option")#facilitate all options, output its value, and click on each option forOptioninchall_options:Print("Value is:%s"% Option.get_attribute ("value") ) Option.click ()

Of course the above is not the most efficient way, selenium provides a class for handling this slelect element. The following methods can be used to understand their use by their method names. You can also use Help () in Ipython to see how a method works and how it is used in detail.

 from Import  = Select (Driver.find_element_by_name ('name')) Select.select_by_index ( Index) Select.select_by_visible_text ("text") Select.select_by_value ( Value)

Think of the following, we may take selenium to do the test, you may need to test all the options, below you can get the known options:

select = Select (Driver.find_element_by_xpath ("XPath"= select.all_selected_ Options

3.3 Submit

When you complete the form form, you will submit the form via submit. One way to do this is by finding the submit button and then letting you respond to the click.

# Suppose your Submit ID is OKdriver.find_element_by_id ("OK"). Click ()

Selenium provides a convenient submit () method that you can complete when you invoke the Submit () method from the form element.

Element.submit ()

3.4 Drag

You can move an element to a certain distance, or you can move an element to another element.

 from Import  = Driver.find_element_by_name ("source"= driver.find_element_by_name ("target"= actionchains (driver) action_chains.drag_and_drop (element, Target). Perform ()

3.5

　　　　

　　

　　

　　

　　

Python crawler Learning (9): Use of selenium