Use selenium to crawl Sina weather

( 1 ) with Selenium Crawling Sina Weather

System environment:

Operating system: MacOS 10.13.6 python:2.7.10

Implement with Virtual environment

First, create a virtual environment:

Mkvirtualenv--python=/usr/bin/python python_2

Second, activate the virtual environment:

Workon python_2

Third, installation Selenium

Pip Install Selenium

Iv. Install the selenium patch file for Firefox:

Brew Install Geckodriver

Five, add a line in the ~/.bash_profile:

Export path= $PATH:/usr/local/cellar/geckodriver/0.22.0/bin

Vi. installation of Beautifulsoup4, lxml, Html5lib:

Pip Install Beautifulsoup4

Pip Install lxml

Pip Install Html5lib

Python code:

#coding: Utf-8

Import sys

Reload (SYS)

sys.setdefaultencoding (' UTF8 ')

from Selenium import webdriver

from Selenium.webdriver.common.keys import keys

Import time, DateTime

from BS4 import BeautifulSoup

Driver = Webdriver. Firefox ()

Driver.get ("http://weather.sina.com.cn")

assert U " Sina " in Driver.title

Elem = driver.find_element_by_id ("Hd_sh_input")

Elem.clear ()

Elem.send_keys (u " Changchun ")

Time.sleep (2)

Elem.send_keys (Keys.return)

Time.sleep (2)

Handles = Driver.window_handles

for handle in handles: # Toggle Window

if handle! = Driver.current_window_handle:

# print ' Switch to second window ', handle

Driver.close () # close the first window

Driver.switch_to.window (handle) # switch to a second window

Html_const = Driver.page_source

Soup = BeautifulSoup (html_const, ' Html.parser ')

Div_tag = Soup.find_all ("div", class_="Blk_fc_c0_i")

for i in Div_tag:

for Tag in I.find_all (True):

if tag[' class '][0] = = ' wt_fc_c0_i_date ':

Print " Date: ", Datetime.date.today (). Strftime ('%Y ') + "-" + tag.string

if tag[' class '][0] = = ' wt_fc_c0_i_temp ':

Print " Temperature: ", tag.string

if tag[' class '][0] = = ' Wt_fc_c0_i_tip ':

Print " Wind: ", tag.string

if tag[' class '][0] = = ' l ' :

Print "PM5 : ", tag.string

if tag[' class '][0] = = ' r ' :

Print " Air quality: ", tag.string

Print "________________"

Driver.close ()

Operation Result:

Date: 2018-09-30

Temperature: 15°c/7°c

Wind: North Wind, class

Pm5:21

Air quality: Excellent

________________

Date: 2018-10-01

Temperature: 15°c/4°c

Wind: northwest of the same class

Pm5:21

Air quality: Excellent

________________

Date: 2018-10-02

Temperature: 19°c/7°c

Wind: Westerly less than level 3

Pm5:40

Air quality: Excellent

________________

Date: 2018-10-03

Temperature: 20°c/8°c

Wind: Southwest wind less than 3 levels

pm5:58

Air quality: Liang

________________

Date: 2018-10-04

Temperature: 21°c/9°c

Wind: Southwest wind less than 3 levels

pm5:57

Air quality: Liang

________________

Date: 2018-10-05

Temperature: 22°c/9°c

Wind: Southwest wind less than 3 levels

Pm5:40

Air quality: Excellent

________________

Use selenium to crawl Sina weather