For example, the configuration of robots.txt and meta name robots on the website

Introduction to robots.txt
Robots.txt is a plain text file in which the website administrator can declare that the website does not want to be accessed by robots, or specify a search engine to include only specified content.
When a search robot (called a search spider) crawls a site, it first checks that the site root directory contains robots.txt. If so, the search robot determines the access range based on the content in the file. If the file does not exist, the search robot crawls the link.
In addition, robots.txt must be placed in the root directory of a site, and all file names must be in lowercase.
Robots.txt writing syntax
First, let's look at a robots.txt example:/robots.txt
The specific content of robots.txt is as follows:
# All robots will spider the domain
User-agent :*
Disallow:
The above text indicates that all search robots are allowed to access blog.2fool.cn. All files under the site.
Specific syntax analysis: # The description is followed by the description; User-agent: The name of the robot to search. *, All search robots; Disallow: The following is the file directory that cannot be accessed.
Next, let me list the specific usage of robots.txt:
Allow access by all robots
User-agent :*
Disallow:
Alternatively, you can create an empty file "/robots.txt" file.
Prohibit all search engines from accessing any part of the website
User-agent :*
Disallow :/
Prohibit all search engines from accessing the website (in the following example, the 01, 02, and 03 Directories)
User-agent :*
Disallow:/01/
Disallow:/02/
Disallow:/03/
Prohibit Access to a search engine (BadBot in the following example)
User-agent: BadBot
Disallow :/
Only access to a search engine is allowed (The Crawler in the following example)
User-agent: Crawler
Disallow:
User-agent :*
Disallow :/
In addition, I think it is necessary to expand the description to introduce robots meta:
The Robots META tag mainly targets specific pages. Like other META tags (such as the language used, page description, and keywords), Robots META tags are also placed in the Syntax of the Robots META tag:
The Robots META tag is case-insensitive. name = "Robots" indicates all search engines. You can enter name = "BaiduSpider" for a specific search engine ". The content part has four Command Options: index, noindex, follow, and nofollow. commands are separated by commas.
The INDEX Command tells the search robot to capture the page;
The FOLLOW command indicates that the search robot can continue crawling along the link on the page;
The default values of the Robots Meta tag are INDEX and FOLLOW, except inktomi. The default values are INDEX and NOFOLLOW.
In this way, there are four combinations:
<Meta name = "ROBOTS" CONTENT = "INDEX, FOLLOW">
<Meta name = "ROBOTS" CONTENT = "NOINDEX, FOLLOW">
<Meta name = "ROBOTS" CONTENT = "INDEX, NOFOLLOW">
<Meta name = "ROBOTS" CONTENT = "NOINDEX, NOFOLLOW">
Where
<Meta name = "ROBOTS" CONTENT = "INDEX, FOLLOW"> <meta name = "ROBOTS" CONTENT = "ALL">;
<Meta name = "ROBOTS" CONTENT = "NOINDEX, NOFOLLOW"> <meta name = "ROBOTS" CONTENT = "NONE">
Currently, a huge number of search engine Robots comply with the robots.txt rules. Currently, the Robots META tag does not support much, but is gradually increasing. For example, GOOGLE, a famous search engine, is fully supported, in addition, GOOGLE also adds the command "archive" to limit whether GOOGLE retains web snapshots. For example:
<Meta name = "googlebot" CONTENT = "index, follow, noarchive">
It indicates that the page in the site is crawled along the link on the page, but the page snapshot is not retained on GOOLGE.