What is robots.txt file?

Search engines use the spider program to automatically access Web pages on the Internet and get web information. When you visit a Web site, spider first checks to see if there is a plain text file called robots.txt under the root domain of the Web site that specifies Spider's crawl range on your site. You can create a robots.txt in your Web site, stating in the file that the site does not want to be included in the search engine or that the specified search engine contains only a specific section.

Note that you need to use the robots.txt file only if your site contains content that you do not want to be included in the search engine. If you want search engines to include everything on your site, do not create robots.txt files.

The robots.txt file should be placed in the root directory of the site. For example, when Spider accesses a Web site (such as http://www.domain.com), it first checks to see if the site exists http://www.domain.com/ robots.txt This file, if Spider finds the file, it will determine the scope of its access rights based on the contents of the file. Note: If other websites link to the banned pages set in your robots.txt file, then these pages may still appear in Baidu's search results, but the content of your Web page will not be crawled, indexed and displayed, Baidu search results show only the other sites on your related pages description. (Source from: Ningbo Network marketing, 863.html ">seo Search engine Optimization Research Center, if you need to reprint, please specify the source.) ）