Introduction to robots.txt Configuration

As we know, search engines all have their own "Search robots" and use these robots to link on the Web page over the network (generally HTTP and SRC links) constantly crawl data to build your own database.

For website administrators and content providers, there are sometimes some website content that they do not want to be crawled by robots. To solve this problem, the robots Development Community provides two methods: robots.txt and therobotsmeta.

Note: It is critical for the search engine to write robots.txt correctly. We try to write statements in the standard format. Otherwise, errors may cause the search engine to fail to crawl the site; we can use the robots.txtdetection tool in Google sitemapto check the existence of the robots.txt file on the website and whether the file is correctly written.

One worker robots.txt

What is robots.txt?

Robots.txt is a plain text file that declares that the website does not want to be accessed by robots. In this way, some or all of the content of the website can be not indexed by the search engine, or the specified search engine can only include the specified content.

When a robot searches for a site, it first checks that the site root directory contains robots.txt. if it finds it, the search robot determines the access range based on the content in the file. If the file does not exist, the search robot crawls along the link.

Robots.txt must be placed in the root directory of a site and all file names must be in lowercase.

Website URL
URL of robots.txt

Http://www.w3.org/
Http://www.w3.org/robots.txt

Http://www.w3.org: 80/
Http://www.w3.org: 80/robots.txt.

2) syntax of robots.txt

The "robots.txt" file contains one or more records separated by empty rows (ending with Cr, CR/NL, or ornl ), in this file, you can use # For annotation. The usage is the same as that in UNIX. The record in this file usually starts with one or more lines of User-Agent, followed by several disallow lines. The details are as follows:

User-Agent:
The value of this item is used to describe the name of the search engine robot. In the "robots.txt" file, if Multiple User-Agent records indicate that multiple robots are restricted by this Protocol, for this file, there must be at least one User-Agent record. If this parameter is set to *, the Protocol is valid for all machines. In the "robots.txt" file, only one record such as "User-Agent: *" can exist. For the name of the search engine robots, seeArticle"Search engine spiderProgramName Daquan"

Disallow:
The value of this item is used to describe a URL that you do not want to access. This URL can be a complete or partial path, any URL starting with disallow will not be accessed by the robot. For example, "disallow:/help" does not allow access by search engines to/help.html and/help/index.html, while "disallow:/help/" allows robot access to/help.html, but cannot access/help/index.html.

If any disallow record is blank, all parts of the website are allowed to be accessed. At least one disallow record is required in the "robots.txt" file. If "robots.txt" is an empty file, the website is open to all search engine robots.

Below are some basic usage of robots.txt:

Prohibit all search engines from accessing any part of the website:
User-Agent :*
Disallow :/

Allow access by all robots
User-Agent :*
Disallow:
Alternatively, you can create an empty file: robots.txt.

Prohibit all search engines from accessing the website (cgi-bin, TMP, and private directories in the following example)
User-Agent :*
Disallow:/cgi-bin/
Disallow:/tmp/
Disallow:/private/

Prohibit Access to a search engine (badbot in the following example)
User-Agent: badbot
Disallow :/

Only allow access to a search engine (webcrawler in the following example)
User-Agent: webcrawler
Disallow:

3. Names of common search engine robots

Name Search Engine URL

Baiduspider http://www.baidu.com

Scooter http://www.altavista.com.

Ia_archiver http://www.alexa.com

Googlebot http://www.google.com

Inktomi slurp http://www.yahoo.com

Fast-webcrawler http://www.alltheweb.com

Slurp http://www.inktomi.com

Msnbot http://search.msn.com

4、robots.txt example

The following is the robots.txt of some famous sites:

Http://www.google.com/robots.txt

Http://www.ibm.com/robots.txt

Http://www.sun.com/robots.txt

Http://www.eachnet.com/robots.txt

Look at a hundred degrees of robots.txt: http://www.baidu.com/robots.txt

Luke's Seo blog's robots.txt file: http://www.bloghuman.com/robots.txt

5、 robots.txt Error

· Reversed the order:
Write errors
User-Agent :*
Disallow: googlebot

Which of the following statements is true:
User-Agent: googlebot
Disallow :/

· Put multiple prohibited commands in one line:
For example, you can mistakenly write
Disallow:/CSS // cgi-bin // images/

Which of the following statements is true?
Disallow:/CSS/
Disallow:/cgi-bin/
Disallow:/images/

· There are a large number of spaces in front of the line
For example
Disallow:/cgi-bin/
Although this is not mentioned in standards, this method is prone to problems.

· Redirect 404 to another page:
When many websites with no robots.txt file configured in the robotnet, the system will automatically redirect 404 to another HTML page. In this case, the robotnet will usually process the HTML page file in the form of a robots.txt file. Although there is no such question, it is best to put a blank robots.txt file under the root directory of the site.

· Use uppercase letters
User-Agent: excite
Disallow:
Although the standard is case-insensitive, the directory and file name should be in lower case ,:
User-Agent: googlebot
Disallow:

· In the syntax, only disallow and no allow!
Incorrect syntax:
User-Agent: baiduspider
Disallow:/John/
Allow:/Jane/

· Forgot the slash/
Incorrect write:
User-Agent: baiduspider
Disallow: CSS

Which of the following statements is true?
User-Agent: baiduspider
Disallow:/CSS/

The following is a small tool dedicated to checking the validity of the robots.txt file:

Http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

Ii. Robots meta tag

1. What is a robots meta tag?

The robots.txt file mainly restricts the access of search engines to the entire site or directory, while the robots meta tag mainly targets specific pages. Like other meta tags (such as the language used, page description, and keywords), robots meta tags are also placed on the page, specifically used to tell the search engine robots how to capture the content of this page.

2. syntax of the robots meta tag:

The robots meta tag is case-insensitive. Name = "Robots" indicates all search engines. You can enter name = "baiduspider" for a specific search engine ". The content part has four Command Options: Index, noindex, follow, and nofollow. commands are separated by commas.

The Index Command tells the search robot to capture the page;

The follow command indicates that the search robot can continue crawling along the link on the page;

The default values of the robots meta tag are index and follow, except Inktomi. The default values are index and nofollow.

It should be noted that the robots.txt and robots meta tags described above restrict the search engine robot (robots) to capture site content only in a way that requires the cooperation of Search Engine robots, not every robots does.

Currently, a huge number of search engine robots comply with the robots.txt rules, but the robotsmeta tag currently does not support much, but is gradually increasing. For example, Google, the famous search engine, fully supports it, and Google also adds an "ARCHIVE" command ", you can restrict whether or not Google retains web snapshots. For example:

It indicates that the page in the site is crawled along the link on the page, but the page snapshot is not retained on goolge.

Example:
# Robots, scram

User-Agent :*
Disallow:/cgi-bin
Disallow:/transcripts
Disallow:/Development
Disallow:/Third
Disallow:/beta
Disallow:/Java
Disallow:/Shockwave
Disallow:/jobs
Disallow:/PR
Disallow:/interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/digest
Disallow:/quicknews
Disallow:/search

User-Agent: Mozilla/3.01 (hotwired-test/0.1)
disallow:/cgi-bin
disallow:/transcripts
disallow: /development
disallow:/Third
disallow:/beta
disallow:/Java
disallow:/Shockwave
disallow: /jobs
disallow:/Pr
disallow:/interactive
disallow:/alt_index.html
disallow:/webmaster_logs
disallow: /newscenter
disallow:/virtual
disallow:/digest
disallow:/quicknews
disallow:/search

User-Agent: slurp
disallow:/cgi-bin
disallow:/transcripts
disallow:/development
disallow: /Third
disallow:/beta
disallow:/Java
disallow:/Shockwave
disallow:/jobs
disallow: /Pr
disallow:/interactive
disallow:/alt_index.html
disallow:/webmaster_logs
disallow:/newscenter
disallow: /virtual
disallow:/digest
disallow:/quicknews
disallow:/search
User-Agent: scooter
disallow: /cgi-bin
disallow:/transcripts
disallow:/development
disallow:/Third
disallow:/beta
disallow: /Java
disallow:/Shockwave
disallow:/jobs
disallow:/Pr
disallow:/interactive
disallow: /alt_index.html
disallow:/webmaster_logs
disallow:/newscenter
disallow:/virtual
disallow:/digest
disallow: /quicknews
disallow:/search

User-Agent: ultraseek
disallow:/cgi-bin
# disallow:/transcripts
disallow:/development
disallow: /Third
disallow:/beta
disallow:/Java
disallow:/Shockwave
disallow:/jobs
disallow: /Pr
disallow:/interactive
disallow:/alt_index.html
disallow:/webmaster_logs
disallow:/newscenter
disallow: /virtual
disallow:/digest
disallow:/quicknews
disallow:/search

User-Agent: smallbear
disallow:/cgi-bin
disallow:/Java
disallow:/images
disallow: /development
disallow:/Third
disallow:/beta
disallow:/webmaster_logs
disallow:/virtual
disallow: /Shockwave
disallow:/transcripts
disallow:/newscenter
disallow:/virtual
disallow:/digest
disallow: /quicknews
disallow:/search
disallow:/alt_index.html
User-Agent: googlebot
disallow:/cgi-bin
disallow: /Java
disallow:/images
disallow:/development
disallow:/Third
disallow:/beta
disallow: /webmaster_logs
disallow:/virtual
disallow:/Shockwave
disallow:/transcripts
disallow:/newscenter
disallow: /virtual
disallow:/digest
disallow:/quicknews
disallow:/search
disallow:/alt_index.html