After several days of study, I summarized the following points: 1. a spider can only capture href (<a href = "Default. aspx "> test </a>) it is best not to include a parameter (<a href =" Default. aspx? Id = 1 "> test </a>) If you do not have to consider the parameter Spider, you need to use the URL to rewrite the URL. the SPIDER will not execute JavaScript. In other words, if onclick is used in tag a, the spider will not catch it. 3. the spider can only capture the page of the get request but not the page of the post request. 4. we hope that all the front-end pages of the webpage will be caught by the spider, but we do not want the backend pages to be caught by the spider. Instead, we only need to know which website is the front-end page and which is the background page. here we need to create "“robots.txt" (robots.txt is a protocol, which is not a command. It is generally best to follow) robots.txt is the first file used by the search engine to search for the website. The following describes the focus of this article. In, I said that the SPIDER will not execute JavaScript. Does it mean that the AJAX effect will not be affected by the spider? catch it? The answer is no talking about Ajax. Let's take a look at a demo Using AJAX