Php web crawler

Compile the web crawler pcntl_fork or swoole_process in php to implement multi-process concurrency. It takes 500 ms to capture each web page, and 200 processes can be opened to capture 400 pages per second.

Curl can capture pages and set cookies to simulate logon.

Simple_html_dom implements page parsing and DOM processing

To simulate a browser, use casperJS. Encapsulate a service interface with swoole extension to call the PHP layer

Here, a crawler system is implemented based on the above technical solutions. it captures tens of millions of pages every day.