Details about the robots.txt and robots meta tags

For website administrators and content providers, there are sometimes some website content that they do not want to be crawled by robots. To solve this problem, the robots Development Community provides two methods: robots.txt and the robots meta tag.

I. robots.txt

1. What is robots.txt?

Robots.txt is a plain text file that declares that the website does not want to be accessed by robots. In this way, some or all of the content of the website can be not indexed by the search engine, or the specified search engine can only include the specified content.

When a robot searches for a site, it first checks that the site root directory contains robots.txt. if it finds it, the search robot determines the access range based on the content in the file. If the file does not exist, the search robot crawls along the link.

Robots.txt must be placed in the root directory of a site and all file names must be in lowercase.

Website URL
URL of robots.txt

Http://www.w3.org/
Http://www.w3.org/robots.txt

Http://www.w3.org: 80/
Http://www.w3.org: 80/robots.txt.

Http://www.w3.org: 1234/
Http://www.w3.org: 1234/robots.txt

Http://w3.org/
Http://w3.org/robots.txt

 

2. robots.txt syntax

The "robots.txt" file contains one or more records separated by empty rows (with Cr, CR/NL, or NL as the terminator). The format of each record is as follows:

"<Field >:< optionalspace> <value> <optionalspace> ".

In this file, you can use # For annotation. The usage is the same as that in UNIX. The record in this file usually starts with one or more lines of User-Agent, followed by several disallow lines. The details are as follows:

User-Agent:

The value of this item is used to describe the name of the search engine robot. In the "robots.txt" file, if Multiple User-Agent records indicate that multiple robots are restricted by this Protocol, for this file, there must be at least one User-Agent record. If this parameter is set to *, the Protocol is valid for all machines. In the "robots.txt" file, only one record such as "User-Agent: *" can exist.

Disallow:

The value of this item is used to describe a URL that you do not want to access. This URL can be a complete or partial path, any URL starting with disallow will not be accessed by the robot. For example, "disallow:/help" does not allow access by search engines to/help.html and/help/index.html, while "disallow:/help/" allows robot access to/help.html, but cannot access/help/index.html.

If any disallow record is blank, it means that all parts of the website are allowed to be accessed. At least one disallow record is required in the "/robots.txt" file. If "/robots.txt" is an empty file, the website is open to all search engine robots.

Below are some basic usage of robots.txt:

L prohibit all search engines from accessing any part of the website:
User-Agent :*
Disallow :/

L allow access by all robots
User-Agent :*
Disallow:
Alternatively, you can create an empty file "/robots.txt" File

L prohibit all search engines from accessing the website (cgi-bin, TMP, and private directories in the following example)
User-Agent :*
Disallow:/cgi-bin/
Disallow:/tmp/
Disallow:/private/

L prohibit access to a search engine (badbot in the following example)
User-Agent: badbot
Disallow :/

L only allow access to a search engine (webcrawler in the following example)
User-Agent: webcrawler
Disallow:

User-Agent :*
Disallow :/

3. Names of common search engine robots

Name Search Engine

Baiduspider:Http://www.baidu.com 

Scooter:Http://www.altavista.com

Ia_archiver:Http://www.alexa.com

GoogleBOT:Http://www.google.com

Fast-webcrawler:Http://www.alltheweb.com

Slurp:Http://www.inktomi.com

Msnbot:Http://search.msn.com 

4. robots.txt example

The following is the robots.txt of some famous sites:

Http://www.cnn.com/robots.txt

Http://www.google.com/robots.txt

Http://www.ibm.com/robots.txt

Http://www.sun.com/robots.txt 

Http://www.eachnet.com/robots.txt

5. Invalid robots.txt Error

L reversed the order:
Write errors
User-Agent :*
Disallow:GoogleBot

Which of the following statements is true:
User-Agent:GoogleBot
Disallow :*

L put multiple prohibited commands in one line:
For example, you can mistakenly write
Disallow:/CSS // cgi-bin // images/

Which of the following statements is true?
Disallow:/CSS/
Disallow:/cgi-bin/
Disallow:/images/

L there are a large number of spaces in front of the line
For example
Disallow:/cgi-bin/
Although this is not mentioned in standards, this method is prone to problems.

L redirect 404 to another page:
When many websites with no robots.txt file configured in the robotnet, the system will automatically redirect 404 to another HTML page. In this case, the robotnet will usually process the HTML page file in the form of a robots.txt file. Although there is no such question, it is best to put a blank robots.txt file under the root directory of the site.

L in uppercase. For example
User-Agent: excite
Disallow:
Although the standard is case-insensitive, the directory and file name should be in lower case:
User-Agent:GoogleBot
Disallow:

L only disallow and no allow!
Incorrect syntax:
User-Agent: baiduspider
Disallow:/John/
Allow:/Jane/

L forgot the slash/
Incorrect write:
User-Agent: baiduspider
Disallow: CSS

Which of the following statements is true?
User-Agent: baiduspider
Disallow:/CSS/

The following is a small tool dedicated to checking the validity of the robots.txt file:

Http://www.searchengineworld.com/cgi-bin/robotcheck.cgi
 

Ii. Robots meta tag

1. What is a robots meta tag?

The robots.txt file mainly restricts the access of search engines to the entire site or directory, while the robots meta tag mainly targets specific pages. Like other meta tags (such as the language used, page description, and keywords), robots meta tags are also placed in the

<HTML>

<Head>

<Title> marketing of the times-professional portal of network marketing </title>

<Meta name = "Robots" con_tent = "index, follow">

<Meta http-equiv = "Content-Type" con_tent = "text/html; charset = gb2312">

<Meta name = "keywords" con_tent = "marketing... ">

<Meta name = "Description" con_tent = "Times marketing network is... ">

<LINK rel = "stylesheet" href = "/public/css.css" type = "text/CSS">

</Head>

<Body>

...

</Body>

</Html>

2. syntax of the robots meta tag:

The robots meta tag is case-insensitive. Name = "Robots" indicates all search engines. You can enter name = "baiduspider" for a specific search engine ". The content part has four Command Options: Index, noindex, follow, and nofollow. commands are separated by commas.

The Index Command tells the search robot to capture the page;

The follow command indicates that the search robot can continue crawling along the link on the page;

The default values of the robots meta tag are index and follow, except Inktomi. The default values are index and nofollow.

In this way, there are four combinations:

<Meta name = "Robots" con_tent = "index, follow">

<Meta name = "Robots" con_tent = "noindex, follow">

<Meta name = "Robots" con_tent = "index, nofollow">

<Meta name = "Robots" con_tent = "noindex, nofollow">

Where

<Meta name = "Robots" con_tent = "index, follow"> can be written

<Meta name = "Robots" con_tent = "all">;

<Meta name = "Robots" con_tent = "noindex, nofollow"> can be written

<Meta name = "Robots" con_tent = "NONE">

It should be noted that the robots.txt and robots meta tags described above restrict the search engine robot (robots) to capture site content only in a way that requires the cooperation of Search Engine robots, not every robots does.

Currently, a huge number of search engines comply with the robots.txt rules. Currently, the robots meta tag does not support much, but is gradually increasing, such as the famous search engine.GoogleAndGoogleAn instruction "ARCHIVE" is added, which can be restricted.GoogleWhether to retain web snapshots. For example:

<Meta name = "googlebot" con_tent = "index, follow, noarchive">

It indicates that the page in the site is crawled along the link on the page, but the page snapshot is not retained on goolge.