Scrapy and selenium combine to crawl dynamic Web pages

1. Install Python (I'm using version 2.7)

2, Installation Scrapy: For details, please refer to http://blog.csdn.net/wukaibo1986/article/details/8167590 (hint, can download the source installed to avoid using PIP install * *)

The solution to the Python extension problem "unable to find Vcvarsall.bat" was encountered during installation: http://blog.csdn.net/ren911/article/details/6448696

3, install Selenium, Https://pypi.python.org/pypi/selenium, note after installation download the server separately, from:http:// Selenium-release.storage.googleapis.com/2.42/selenium-server-standalone-2.42.0.jar, then run (command line run, need to install Java runtime Environment (JRE)) Java-jar Selenium-server-standalone-2.42.0.jar

4, Construction Scrapy Project, reference http://www.searchtb.com/2011/01/an-introduction-to-crawler.html?spm=0.0.0.0.HrlxVl

5, Scrapy and selenium combined crawl Dynamic Web page: https://github.com/tmslav/google_adwords_scrape/blob/master/adwords_bot.py

7. Python Selenium Socket ERROR-[Errno] Connection refused:http://stackoverflow.com/questions/12913991/ Python-selenium-socket-error-errno-61-connection-refused

8. Selenium:firefoxprofile exception Can ' t load the profile:https://github.com/tmslav/google_adwords_scrape/blob/ master/adwords_bot.py

Scrapy and selenium combine to crawl dynamic Web pages