Crawling dynamic pages with PYTHON+PHANTOMJS

Some time ago the company needed to crawl some Web pages for data usage. But the main data in the page is Ajax load, the traditional method of fetching is not the data. Later found on the Internet phantomjs, in the absence of the interface to run JS, rendering the DOM. It 's convenient to use this tool to grab the data from the AJAX load.

system Environment: CentOS release 6.5 (Final)

PHANTOMJS version: 1.9.8

• phantomjs Crawl loads the full DOM structure. Speaking of Phantomjs how to pass data to handlers, I see a lot of people on the web are writing a local file, and then the specific handler then reads that file for processing. Feeling this way is too troublesome, simply print the data to the standard output, and the handler reads the data from the standard output.

• get data in Python. And then we started to deal with it. The specific processing logic is not shown.

Crawling dynamic pages with PYTHON+PHANTOMJS