Python Learning notes-crawler crawling strategies

1 Depth-First algorithm

2 breadth/Width priority policy

3 Complete binary Tree traversal results

Results of depth-first traversal: [1, 3, 5, 7, 9, 4, 12, 11, 2, 6, 14, 13, 8, 10]

Results of breadth-first traversal: [1, 3, 2, 5, 4, 6, 8, 7, 9, 12, 11, 14, 13, 10]

4 How to combine crawl strategies in practice

(1) In general, important pages are close to the entrance site;

(2) Breadth/width preference is advantageous to multi-crawler parallel cooperation;

(3) Consider combining depth and breadth/width to achieve a crawl strategy: prioritize breadth first and limit depth to depths.

The flow of 51 general purpose crawlers is as follows

(1) Setting the seed site, width and depth

(2) a downloaded queue to record all URLs that have completed the download

(3) Implement a function that obtains the contents of the current URL and all the external links

(4) Recursively call this function to traverse the Web site

(5) Error log processing

Python Learning notes-crawler crawling strategies