Have php web crawlers developed similar programs? Can give some advice. The functional requirement is to automatically obtain relevant data from the website and store the data in the database. PHP web crawler database industry data php web crawler
Have you ever developed a similar program? Can give some advice. The functional requirement is to automatically obtain relevant data from the website and store the data in the database.
PHP web crawler database industry data
Share:
------ Solution --------------------
Curl crawls the target website, obtains the corresponding data through regular expressions or DOM, and then stores the database or file.
There is nothing that is too difficult. You need to consider the following:
Crawling policy (only crawling specific domain names, depth first or breadth first ).
Crawling efficiency (multi-thread crawling and how to allocate crawling tasks)
........
