Python Crawler Selenium Library Learning

One, automated testing tools, support a variety of browsers, solve JS rendering problem two, installation

PIP3 Install Selenium

Third, Operation introduction (because it is to learn other people's courses in order to respect intellectual property rights, part of the code will not show) 1 Drive browser

Browser = Webdriver. Chrome ()

Try
Browser.get (' www.sina.com ') #上网

2 finding elements

One way:

Browser.find_element_by_name ()
Browser.find_element_by_class_name ()
BROWSER.FIND_ELEMENT_BY_ID ()
Browser.find_element_by_xpath ()
Browser.find_element_by_tag_name ()

Wait a minute

The second method:

Browser.find_element (by.name, ' search ')

This find_element is equivalent to BS4 inside the find, if you want to use Find_all, you need to use find_elements

There are two ways to do this, and I'll write one here.

Browser.find_elements (by.name, ' search ')

3 Interactive operations

input = Browser.find_element_by_class_name (' search-combobox-input ')
Input.send_keys (' mechanical keyboard ')
Time.sleep (3)
Input.clear ()
Input.send_keys (' IPad ')
Time.sleep (3)
button = Browser.find_element_by_class_name (' Btn-search ')
Button.Click ()

For more information, see the official documentation:

Http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webelement

To execute the action chain:

It doesn't feel like much, see the official documentation.

Http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.common.action_chains

4 Get Property Display Properties

input = Browser.find_element_by_class_name (' search-combobox-input ')

Print (Input.get_attribute (' Aria-combobox '))

This returns the attribute ' list ' of the Aria-combobox

Display text

Browser = Webdriver. Chrome ()
Try
url = ' Https://www.zhihu.com/explore '
Browser.get (URL)
Answers = Browser.find_elements_by_class_name (' zh-summary ')
For answer in answers:
Print (Answer.text)
Finally
Browser.close ()

This shows the text that knows the answer in the discovery.

Get ID, location, label name, size

Browser = Webdriver. Chrome ()
Try
url = ' Https://www.zhihu.com/explore '
Browser.get (URL)
Answers = Browser.find_elements_by_class_name (' zh-summary ')
For answer in answers:

Print (answer.location)
Print (Answer.tag_name)
Print (answer.size)

Finally
Browser.close ()

Frame

In some Web pages that use frames:

Browser.switch_to.frame (' name ')
Browser.switch_to.parent_frame (' name ')

5 Waiting

Sometimes the network needs to wait for the AJAX request to load, so it needs to wait for the page to load

Implicit wait

When the load page does not show the element we need to find, it implicitly waits for a period of time, if the element is found to return directly

Browser = Webdriver. Chrome ()

Browser.implicitly_wait (10)

url = ' Https://www.zhihu.com/explore '
Browser.get (URL)

Explicit wait

An explicit wait is the time that is required if a specific condition is not met

Browser = Webdriver. Chrome ()
url = ' Https://www.zhihu.com/explore '
Browser.get (URL)
wait = webdriverwait (browser,10)
Intput = Wait.until (ec.presence_of_element_located ((by.id, ' Q ')))
button = Wait.until (Ec.element_to_be_clickable ((By.css_selector, ' * * * ')))

Conditions for an explicit wait see the official document: Http://selenium-python.readthedocs.io/api.html#selenium.webdriver.support.expected_ Conditions.element_located_selection_state_to_be

Title_is title is a content
Title_contains Title contains a content
Presence_of_all_elements_located y element is loaded, passed in the locating tuple, as (by.id, ' P ')
visibility_of element visible, incoming locator tuple

Text_to_be_present_in_element an element literal contains a literal
Text_to_be_present_in_element_value An element value contains a file
Element_to_be_clickable an element can be clicked

Wait a minute

6 Browser forward and backward

Browser.forward ()
Browser.back ()

7Cookies

Cookies can be viewed, added and deleted

url = ' Https://www.zhihu.com/explore '
Browser.get (URL)
Print (Browser.get_cookies ())
Browser.add_cookie ({' name ': ' Dai ', ' value ': ' 123 '}) #一定要记得有value否则会报错
Print (Browser.get_cookies ())
Browser.delete_all_cookies ()
Print (Browser.get_cookies ())

Python Crawler Selenium Library Learning