Site configuration robots.txt file for your reference

What is a robots file?

Search engine through a program robot (also known as Spider), automatic access to Web pages on the Internet and get web information.
You can create a plain text file robots.txt in your Web site, in which you declare the part of the site that you do not want to be robot, so that some or all of the content of the site is not included in the search engine, or the specified search engine contains only the specified content.
Where are the robots.txt files?
The robots.txt file should be placed in the root directory of the site. For example, when a robot accesses a Web site (such as http://www.ithov.com), it first checks the site for the existence of the Http://www.ithov.com/robots.txt file, and if it finds the file, it will determine the scope of its access according to the contents of the file.

robots.txt file format?

The "robots.txt" file contains one or more records that are separated by a blank line (with CR,CR/NL, or NL as The Terminator), and each record is formatted as follows:
"<field>:<optionalspace><value><optionalspace>".
You can use # to annotate in this file, using the same method as in Unix. The records in this file usually start with one or more lines of user, followed by a number of disallow lines, as detailed below:
User:
The value of the item is used to describe the name of the search engine robot, and in the "robots.txt" file, if there are multiple user records stating that a plurality of robot will be restricted by the protocol, there must be at least one user record for the file. If the value of the item is set to *, the protocol is valid for any robot, and in the "robots.txt" file, there can be only one record for "user-agent:*".
Disallow:
The value of the item is used to describe a URL that you do not want to be visited, which can be a complete path or partial, and any URL that starts with disallow will not be accessed by robot. For example, "Disallow:/help" does not allow search engine access for both/help.html and/help/index.html, while "disallow:/help/" allows robot to access/help.html, not access/help/ Index.html. Any disallow record is empty, indicating that all parts of the site are allowed to be accessed, and that in the "/robots.txt" file, there must be at least one disallow record. If "/robots.txt" is an empty file, the site is open to all search engine robot.

robots.txt Grammar Tutorial

Use a few of the most common cases to illustrate directly:
1. Allow all se to be included in this site: robots.txt is empty, do not write anything.
2. To prohibit certain catalogues of all SE included websites:
User: *
Disallow:/directory Name 1/
Disallow:/directory Name 2/
Disallow:/directory Name 3/

3. Prohibit a certain SE included in this site, such as the prohibition of Baidu:
User-agent:baiduspider
Disallow:/

4. Prohibit all SE included in this site:
User: *
Disallow:/

5. Join the Sitemap.xml path, for example:
Sitemap:http://www.ithov.com/sitemap.xml

Reference: Common search engine robot robots name
Name Search engine
Baiduspider http://www.baidu.com
Googlebot http://www.google.com
MSNBOT http://search.msn.com
Ia_archiver http://www.alexa.com

A tool for writing robots.txt online.
Http://www.clickability.co.uk/robotstxt.html

Finally, note that the robots.txt file name must be lowercase and placed in the root directory of the site, such as: Http://www.ithov.com/robots.txt

According to the Site directory structure, there are some differences, the following robots.txt file for your reference to use:

Seriously despise reprint documents, remove copyright information! Previously published an article by some Web site changed beyond recognition, the copyright did not leave anything!

# robots.txt file Start

# Exclude Files from all Robots:

User: *
Disallow:/article/
Disallow:/master/
Disallow:/ebook/
Disallow:/code/
Disallow:/soft/
Disallow:/video/
Disallow:/mov/
Disallow:/user/
Disallow:/admin/
Sitemap:http://www.ithov.com/sitemap_index.xml

# End Robots.txt File

As to why the root directory so write disallow:/article/Way, we can carefully observe the structure of my site access, you can understand the reasons why! Of course, still need more careful observation search engine collection, for their own site configuration a correct robots.txt is the key.

Author: Sword Tree Source: It square information Network http://www.ithov.com Copyright All welcome the media to each other, please specify the author and the source!