Chapter 2 macro view of crawler Key Points of spider

Chapter 2: Looking at the macro view, the crawler Key Points of spider web information is heterogeneous and dynamic. Due to time, storage, and bandwidth restrictions, it is impossible to collect all webpages, A good collection strategy is limited collection of important webpages. The importance of web pages depends on the different applications for which information is collected, so that different policies can be adopted for information collection. At present, this issue is still inconclusive. The importance of web pages is generally determined based on the following indicators:
1) The webpage has a high inbound level, that is, the number of times it is referenced;
2) The parent page of the webpage has a high inbound level;
3) The webpage has multiple image IP addresses;
4) The Directory depth of the web page is small, which is easy for users;
5) webpage content has a high information entropy;
6) webpage updates are frequently performed;
When the search engine starts to work, the above parameters are unknown, which is a progressive optimization process. Important pages are not necessarily the objects to be crawled first. You also need to take into account other parameters that affect performance and perform optimization step by step.