How to write the php crawler link pool and link scheduling?

Link pool function: 1. Store links; 2. deduplicate links; 3. set priority for links. Practice solutions: 1. stored in the database; 2. redis; 3. Memory set; 4. queue. How to write the connection pool and capture and Dispatch code?

Link pool function: 1. Store links; 2. deduplicate links; 3. set priority for links.
Practice solutions: 1. stored in the database; 2. redis; 3. Memory set; 4. queue.

How to write the connection pool and capture and Dispatch code?

Reply content:

Link pool function: 1. Store links; 2. deduplicate links; 3. set priority for links.
Practice solutions: 1. stored in the database; 2. redis; 3. Memory set; 4. queue.

How to write the connection pool and capture and Dispatch code?

Get all the links to be crawled first, and then put the crawling actions of each link in the queue. We recommend using beanstalk, and then the queue executes these actions.