WebMagic User Manual

https://my.oschina.net/flashsword/blog/180623

Focus

Seleniumdownloader

For some JavaScript dynamically loaded Web pages, just use the HTTP emulation download tool, and cannot fetch the content of the page. There are two ways of thinking in this regard: one is to cobwebs, to analyze the logic of JS, then to reproduce it with a reptile, and the other is to have a browser built in to get the last loaded page directly. The Webmagic-selenium package integrates selenium to seleniumdownloader and can be crawled directly into the dynamically loaded page. Use selenium need to install some native tools, specific steps can refer to the author's blog post using selenium to crawl dynamically loaded pages

WebMagic User Manual