Chrome browser crawler crashes, what to do? Python + Selenium + chrome + headless mode __python

Chrome Browser crawler crashes, how to do. Python + Selenium + chrome + headless mode 1. Background

When crawling data using the Selenium + Chrome browser rendering mode, if there are too many concurrent tasks, or if the crawler is running for a long time, it is easy to crash the browser, as follows:

This is typically caused by excessive resource consumption (it is said that the Chrome browser has a memory leak). So how to solve this problem. In this case, we first think of using a Phantomjs browser, but the PHANTOMJS is now in disrepair, followed by no one, with many new features that support is not good enough. But the good news is that Google has added a new feature to chrome this year: Headless mode, which allows us to crawl data using chrome without interface, with less resources and faster. And thankfully, Google's team is committed to maintaining it ... 2. Environment System: Win7 MongoDB 3.4.6 python 3.6.1 ide:pycharm installed Chrome browser (63.0.3239.132 (official version) 32-bit) selenium 3.7.0

Configure Good Chromedriver v2.34

Special Note:

Because headless mode is a new feature, only advanced versions can be used, not forward compatible, so the version of the Chrome browser and chromedriver is required: 
1. For Chrome: 
   Linux, The UNIX system requires a Chrome browser >= 
   The Windows system requires a Chrome browser >=
2. Chromedriver version matches the Chrome browser:
   This part of the reference article: http:/ /blog.csdn.net/zwq912318834/article/details/78550666

3. Code

From selenium import Webdriver to
selenium.webdriver.common.by import by from
Selenium.webdriver.support.ui Import webdriverwait from
selenium.webdriver.support import expected_conditions as EC
from Selenium.webdriver.common.action_chains import actionchains from
Selenium.webdriver.common.keys import keys

chrome_options = Webdriver. Chromeoptions ()
# Using headless no-interface browser mode
chrome_options.add_argument ('--headless ')
chrome_options.add_ Argument ('--disable-gpu ')

# Launch Browser, get web page source code
browser = webdriver. Chrome (chrome_options=chrome_options)
mainurl = "https://www.taobao.com/"
browser.get (Mainurl)
Print (f "browser text = {Browser.page_source}")
Browser.quit ()

You can see that there is no browser interface anymore. Generally we develop the order is: 1. First take off this headless parameter, develop in the browser interface; 2. After the stable development, the headless parameters are added and tested to see if it is stable 3. If you are stable, deploy again.