Python crawler notes: phantomjs + selenium content, pythonphantomjs

Source: Internet
Author: User

Python crawler notes: phantomjs + selenium content, pythonphantomjs

For general websites, beautifulsoup using python can be crawled, but in the face of some websites that need to execute JavaScript on the page, you can use phantomjs + selenium to crawl data. When I was studying, I also encountered this kind of problem.

My Use Case website is China weather network (http://www.weather.com.cn/weather40d/101020100.shtml ).

What I want to crawl is the highest temperature data of every day in Shanghai's 40-day weather. Therefore, first I use the general method to crawl:

from bs4 import BeautifulSoupfrom urllib.request import urlopenhtml = urlopen('http://www.weather.com.cn/weather40d/101020100.shtml')html_parse = BeautifulSoup(html)temp = html_parse.findAll("span",{"class":"max"})print(temp)

However, print (temp) outputs only labels: [<span class = "max"> </span>, <span class = "max"> </span> ...... <span class = "max"> </span>]

Therefore, I decided that data must be obtained only after javascript Execution. So I used phantomjs + selenium to obtain this type of data. The Code is as follows:

from bs4 import BeautifulSoupfrom selenium import webdriverimport timedriver = webdriver.PhantomJS(executable_path='F:\\python\\phantomjs-2.1.1-windows\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')driver.get("http://www.weather.com.cn/weather40d/101020100.shtml")time.sleep(3)pageSource = driver.page_sourcehtml_parse = BeautifulSoup(pageSource)temp = html_parse.findAll("span",{"class":"max"})print(temp)

This Code creates a new selenium WebDriver, first load the page with WebDriver, so we give it 3 seconds (time. sleep (3), since I personally prefer beautifulsoup, And the page_source function of WebDriver can return the source code string of the page, therefore, I used line 8 and 9 code to return to parsing the page content with the Beautifulsoup we are familiar. The final running result of this program is: [<span class = "max"> 9 </span>, <span class = "max"> 9 </span> ...... <span class = "max"> 12 </span>, <span class = "max"> 12 </span>, <span class = "max"> </span>, <span class = "max"> </span>, <span class = "max"> </span>], the data can be basically obtained.

Although this example is relatively simple, the basic idea of the so-called eternal changes is that we need to continue to learn more advanced technologies.

If there are any mistakes in this article, you are welcome to point out, learn together, and make progress together.

 

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.