Chrome browser crawler crashes, what to do? Python + Selenium + chrome + headless mode __python

Source: Internet
Author: User
Chrome Browser crawler crashes, how to do. Python + Selenium + chrome + headless mode 1. Background

When crawling data using the Selenium + Chrome browser rendering mode, if there are too many concurrent tasks, or if the crawler is running for a long time, it is easy to crash the browser, as follows:

This is typically caused by excessive resource consumption (it is said that the Chrome browser has a memory leak). So how to solve this problem. In this case, we first think of using a Phantomjs browser, but the PHANTOMJS is now in disrepair, followed by no one, with many new features that support is not good enough. But the good news is that Google has added a new feature to chrome this year: Headless mode, which allows us to crawl data using chrome without interface, with less resources and faster. And thankfully, Google's team is committed to maintaining it ... 2. Environment System: Win7 MongoDB 3.4.6 python 3.6.1 ide:pycharm installed Chrome browser (63.0.3239.132 (official version) 32-bit) selenium 3.7.0

Configure Good Chromedriver v2.34

Special Note:

Because headless mode is a new feature, only advanced versions can be used, not forward compatible, so the version of the Chrome browser and chromedriver is required: 
1. For Chrome: 
   Linux, The UNIX system requires a Chrome browser >= 
   The Windows system requires a Chrome browser >=
2. Chromedriver version matches the Chrome browser:
   This part of the reference article: http:/ /blog.csdn.net/zwq912318834/article/details/78550666
3. Code
From selenium import Webdriver to
selenium.webdriver.common.by import by from
Selenium.webdriver.support.ui Import webdriverwait from
selenium.webdriver.support import expected_conditions as EC
from Selenium.webdriver.common.action_chains import actionchains from
Selenium.webdriver.common.keys import keys

chrome_options = Webdriver. Chromeoptions ()
# Using headless no-interface browser mode
chrome_options.add_argument ('--headless ')
chrome_options.add_ Argument ('--disable-gpu ')

# Launch Browser, get web page source code
browser = webdriver. Chrome (chrome_options=chrome_options)
mainurl = "https://www.taobao.com/"
browser.get (Mainurl)
Print (f "browser text = {Browser.page_source}")
Browser.quit ()
You can see that there is no browser interface anymore. Generally we develop the order is: 1. First take off this headless parameter, develop in the browser interface; 2. After the stable development, the headless parameters are added and tested to see if it is stable 3. If you are stable, deploy again.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.