Seo robots.txt setup tutorial

About the syntax and function of robots. txt

As we know, search engines all have their own "Search robots" and use these robots to link on the Web page over the network (generally HTTP and SRC links) constantly crawl data to build your own database.

For website administrators and content providers, there are sometimes some website content that they do not want to be crawled by robots. To solve this problem, the robots Development Community provides two methods: robots.txt and the robots meta tag.

I. robots.txt

1. What is robots.txt?
Robots.txt is a plain text file that declares that the website does not want to be accessed by robots. In this way, some or all of the content of the website can be not indexed by the search engine, or the specified search engine can only include the specified content. When a robot searches for a site, it first checks that the site root directory contains robots.txt. if it finds it, the search robot determines the access range based on the content in the file. If the file does not exist, the search robot crawls along the link.

Robots.txt must be placed in the root directory of a site and all file names must be in lowercase.
Website URL
URL of robots.txt
Http://www.ali173.com/
Http://www.ali173.com/robots.txt

Http://www.ali173.com: 80/
Http://www.ali173.com: 80/robots.txt.

Http://www.ali173.com: 1234/
Http://www.ali173.com: 1234/robots.txt

Http://ali173.com/
Http://ali173.com/robots.txt

2. robots.txt syntax

The "robots.txt" file contains one or more records separated by empty rows (with Cr, CR/NL, or NL as the terminator). The format of each record is as follows:

":".

In this file, you can use # For annotation. The usage is the same as that in UNIX. The record in this file usually starts with one or more lines of User-Agent, followed by several disallow lines. The details are as follows:

User-Agent:

The value of this item is used to describe the name of the search engine robot. In the "robots.txt" file, if Multiple User-Agent records indicate that multiple robots are restricted by this Protocol, for this file, there must be at least one User-Agent record. If this parameter is set to *, the Protocol is valid for all machines. In the "robots.txt" file, only one record such as "User-Agent: *" can exist.

Disallow:

The value of this item is used to describe a URL that you do not want to access. This URL can be a complete or partial path, any URL starting with disallow will not be accessed by the robot. For example, "disallow:/help" does not allow access by search engines to/help.html and/help/index.html, while "disallow:/help/" allows robot access to/help.html, but cannot access/help/index.html.

If any disallow record is blank, it means that all parts of the website are allowed to be accessed. At least one disallow record is required in the "/robots.txt" file. If "/robots.txt" is an empty file, the website is open to all search engine robots.

Below are some basic usage of robots.txt:

L prohibit all search engines from accessing any part of the website:
User-Agent :*
Disallow :/

L allow access by all robots
User-Agent :*
Disallow:
Alternatively, you can create an empty file "/robots.txt" File

L prohibit all search engines from accessing the website (cgi-bin, TMP, and private directories in the following example)
User-Agent :*
Disallow:/cgi-bin/
Disallow:/tmp/
Disallow:/private/

L prohibit access to a search engine (badbot in the following example)
User-Agent: badbot
Disallow :/

L only allow access to a search engine (webcrawler in the following example)

User-Agent: webcrawler
Disallow:

User-Agent :*
Disallow :/

3. Names of common search engine robots

Name Search Engine

Baiduspider http://www.baidu.com
Scooter http://www.altavista.com.
Ia_archiver http://www.alexa.com
Googlebot http://www.google.com
Fast-webcrawler http://www.alltheweb.com
Slurp http://www.inktomi.com
Msnbot http://search.msn.com

4. robots.txt example

The following is the robots.txt of some famous sites:
Http://www.cnn.com/robots.txt
Http://www.google.com/robots.txt
Http://www.ibm.com/robots.txt
Http://www.sun.com/robots.txt
Http://www.eachnet.com/robots.txt

5. Invalid robots.txt Error

L reversed the order:
Write errors
User-Agent :*
Disallow: googlebot

Which of the following statements is true:
User-Agent: googlebot
Disallow :*

L put multiple prohibited commands in one line:
For example, you can mistakenly write
Disallow:/CSS // cgi-bin // images/

Which of the following statements is true?
Disallow:/CSS/
Disallow:/cgi-bin/
Disallow:/images/

L there are a large number of spaces in front of the line
For example
Disallow:/cgi-bin/
Although this is not mentioned in standards, this method is prone to problems.

L redirect 404 to another page:
When many websites with no robots.txt file configured in the robotnet, the system will automatically redirect 404 to another HTML page. In this case, the robotnet will usually process the HTML page file in the form of a robots.txt file. Although there is no such question, it is best to put a blank robots.txt file under the root directory of the site.

L in uppercase. For example
User-Agent: excite
Disallow:
Although the standard is case-insensitive, the directory and file name should be in lower case:
User-Agent: googlebot
Disallow:

L only disallow and no allow!
Incorrect syntax:
User-Agent: baiduspider
Disallow:/John/
Allow:/Jane/

L forgot the slash/
Incorrect write:
User-Agent: baiduspider
Disallow: CSS

Which of the following statements is true?
User-Agent: baiduspider
Disallow:/CSS/

Ii. Robots meta tag

1. What is a robots meta tag?

The robots.txt file mainly restricts the access of search engines to the entire site or directory, while the robots meta tag mainly targets specific pages. Like other meta tags (such as the language used, page description, and keywords), robots meta tags are also placed on the page, specifically used to tell the search engine robots how to capture the content of this page.
2. syntax of the robots meta tag:

The robots meta tag is case-insensitive. Name = "Robots" indicates all search engines. You can enter name = "baiduspider" for a specific search engine ". The content part has four Command Options: Index, noindex, follow, and nofollow. commands are separated by commas.

The Index Command tells the search robot to capture the page;

The follow command indicates that the search robot can continue crawling along the link on the page;

The default values of the robots meta tag are index and follow, except Inktomi. The default values are index and nofollow.

It should be noted that the robots.txt and robots meta tags described above restrict the search engine robot (robots) to capture site content only in a way that requires the cooperation of Search Engine robots, not every robots does.

Currently, a huge number of search engine robots comply with the robots.txt rules. Currently, the robots meta tag does not support much, but is gradually increasing. For example, Google, a famous search engine, is fully supported, in addition, Google also adds the command "ARCHIVE" to limit whether Google retains web snapshots. For example:

Captures the pages on the site and crawls them along the links on the page, but does not keep the snapshots of the pages on goolge.

Example:
# Robots, scram

User-Agent :*
Disallow:/cgi-bin
Disallow:/transcripts
Disallow:/Development
Disallow:/Third
Disallow:/beta
Disallow:/Java
Disallow:/Shockwave
Disallow:/jobs
Disallow:/PR
Disallow:/interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/digest
Disallow:/quicknews
Disallow:/search

User-Agent: Mozilla/3.01 (hotwired-test/0.1)
Disallow:/cgi-bin
Disallow:/transcripts
Disallow:/Development
Disallow:/Third
Disallow:/beta
Disallow:/Java
Disallow:/Shockwave
Disallow:/jobs
Disallow:/PR
Disallow:/interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/digest
Disallow:/quicknews
Disallow:/search

User-Agent: slurp
Disallow:/cgi-bin
Disallow:/transcripts
Disallow:/Development
Disallow:/Third
Disallow:/beta
Disallow:/Java
Disallow:/Shockwave
Disallow:/jobs
Disallow:/PR
Disallow:/interactive
Disallow:/alt_index.html
Disallow:/webmaster_logs
Disallow:/newscenter
Disallow:/virtual
Disallow:/digest
Disallow:/quicknews
Disallow:/search

For example, we have written DZ programs. You should be able to understand them according to the above instructions.

#
# Robots.txt for discuz! Board
# Version 5.5.0
#

User-Agent :*

Disallow:/admin/
Disallow:/API/
Disallow:/Attachments/
Disallow:/customavatars/
Disallow:/images/
Disallow:/forumdata/
Disallow:/include/
Disallow:/ipdata/
Disallow:/templates/
Disallow:/plugins/
Disallow:/mspace/
Disallow:/WAP/
Disallow: admincp. php
Disallow: Ajax. php
Disallow: Digest. php
Disallow: logging. php
Disallow: Member. php
Disallow: memcp. php
Disallow: Misc. php
Disallow: My. php
Disallow: PM. php
Disallow: Post. php
Disallow: Register. php
Disallow: RSS. php
Disallow: search. php
Disallow: seccode. php
Disallow: topicadmin. php