Phantomjs captures the rendered JS webpage (Python code), phantomjspython

Phantomjs captures the rendered JS webpage (Python code), phantomjspython

Recently, a website needs to be crawled, but the pages are generated after JS rendering. The common crawler framework is not fixed, so I thought of using Phantomjs to build a proxy.

Python calls Phantomjs and it seems that there are no ready-made third-party libraries (if any, please let me know). After walking around, we found that only pyspider provided the ready-made solution.

After a simple trial, I feel that pyspider is more like a crawler tool for beginners. It is like a mother-in-law, sometimes meticulous, and sometimes chattering. Lightweight gadgets should be more popular. I am also selfish and can use them with my favorite BeautifulSoup, instead of learning PyQuery (pyspider is used to parse HTML ), it does not have to endure the bad experience of writing Python in a browser ).

So it took an afternoon to split the part of pyspider implementing the Phantomjs proxy into a small crawler module. I hope you will like it (thanks to binux !).

Preparations

Of course you need Phantomjs! (In Linux, it is best to use the supervisord daemon. You must keep Phantomjs In the Enabled state when capturing it)
Start with phantomjs_fetcher.js in the project path: phantomjs phantomjs_fetcher.js [port]
Install tornado dependencies (the httpclient module of tornado is used)

Calling is super simple

From tornado_fetcher import Fetcher # create a crawler> fetcher = Fetcher (user_agent = 'phantomj', # simulate the browser's User-Agent phantomjs_proxy = 'HTTP: // localhost: 100 ', # phantomjs address poolsize = 10, # maximum number of httpclient async = False # synchronous or asynchronous) # Start to connect to Phantomjs code and render JS! >>> Fetcher. fetch (url) # execute an additional JS script after the rendering is successful (use the function to pack it!) >>> Fetcher. fetch (url, js_script = 'function () {setTimeout ("window. scrollTo (1000 )')

Code https://github.com/2shou/PhantomjsFetcher