Python crawler notes: phantomjs + selenium content, pythonphantomjs
For general websites, beautifulsoup using python can be crawled, but in the face of some websites that need to execute JavaScript on the page, you can use phantomjs + selenium to crawl data. When I was studying, I also encountered this kind of problem.
My Use Case website is China weather network (http://www.weather.com.cn/weather40d/101020100.shtml ).
What I want to crawl is the highest temperature data of every day in Shanghai's 40-day weather. Therefore, first I use the general method to crawl:
from bs4 import BeautifulSoupfrom urllib.request import urlopenhtml = urlopen('http://www.weather.com.cn/weather40d/101020100.shtml')html_parse = BeautifulSoup(html)temp = html_parse.findAll("span",{"class":"max"})print(temp)
However, print (temp) outputs only labels: [<span class = "max"> </span>, <span class = "max"> </span> ...... <span class = "max"> </span>]
Therefore, I decided that data must be obtained only after javascript Execution. So I used phantomjs + selenium to obtain this type of data. The Code is as follows:
from bs4 import BeautifulSoupfrom selenium import webdriverimport timedriver = webdriver.PhantomJS(executable_path='F:\\python\\phantomjs-2.1.1-windows\\phantomjs-2.1.1-windows\\bin\\phantomjs.exe')driver.get("http://www.weather.com.cn/weather40d/101020100.shtml")time.sleep(3)pageSource = driver.page_sourcehtml_parse = BeautifulSoup(pageSource)temp = html_parse.findAll("span",{"class":"max"})print(temp)
This Code creates a new selenium WebDriver, first load the page with WebDriver, so we give it 3 seconds (time. sleep (3), since I personally prefer beautifulsoup, And the page_source function of WebDriver can return the source code string of the page, therefore, I used line 8 and 9 code to return to parsing the page content with the Beautifulsoup we are familiar. The final running result of this program is: [<span class = "max"> 9 </span>, <span class = "max"> 9 </span> ...... <span class = "max"> 12 </span>, <span class = "max"> 12 </span>, <span class = "max"> </span>, <span class = "max"> </span>, <span class = "max"> </span>], the data can be basically obtained.
Although this example is relatively simple, the basic idea of the so-called eternal changes is that we need to continue to learn more advanced technologies.
If there are any mistakes in this article, you are welcome to point out, learn together, and make progress together.