PHP Writing web crawler

• Pcntl_fork or swoole_process implements multi-process concurrency. The crawl time per page is 500ms, open 200 processes, can achieve 400 pages per second crawl.
  

• Curl implements a page crawl, setting a cookie to enable a simulated login
  

• Simple_html_dom implementing page parsing and DOM processing

• If you want to emulate a browser, you can use Casperjs. Encapsulating a service interface with the swoole extension for PHP layer invocation

In the multi-play network here a set of crawler system is based on the above-mentioned technical solution, will crawl tens of millions of pages per day.

PHP Writing web crawler