Personal experience, search engine guide robots.txt file Tips

Trust you webmaster are dynamic CMS Station, then, the site root directory should have a template template folder, if you do not want to let search engines Crawl template Template folder under the template file, then write robots.txt file should pay attention to a point: (Very important)

Example one: "Disallow:/template"
Refers to/help.html and/template/index.html are not allowed search engine spiders crawl.
Case two: "disallow:/template/"
It means to allow search engine spiders to crawl/template.html, and not crawl/template/index.html.

Robots.txt is a plain text file in which http://www.aliyun.com/zixun/aggregation/8984.html "> webmaster can declare parts of the site that they do not want to be accessed by robots, or the specified search engine only contains the specified content.

When a search robot (some called a search spider) accesses a site, it first checks to see if there is a robots.txt in the root directory of the site, and if so, the search robot will follow the contents of the file to determine the scope of the access; If the file does not exist, the search robot will crawl along the link.

In addition, robots.txt must be placed in the root directory of a site, and the file name must be all lowercase.

Robots.txt Writing Grammar

First, let's look at a robots.txt example: Http://www.brhnn.com/robots.txt

Visit the above specific address, we can see the specific contents of robots.txt are as follows:

# Robots.txt file from http://www.seovip.cn
# All robots would spider the domain

User: *
Disallow:

The above text is meant to allow all search bots to access all the files under the www.brhnn.com site.

The specific grammatical analysis: in which the following text is the description information; User: The name of the search robot, followed by *, refers to all the search robot; Disallow: A file directory that is not allowed to be accessed later.

Below, I'll enumerate some specific uses of robots.txt:

Allow all robot access

User: *
Disallow:

Or you can build an empty file "/robots.txt" files

Prohibit all search engines from accessing any part of the site

User: *
Disallow:/

Prohibit all search engines from accessing several parts of the site (01, 02, 03 directories in the following example)

User: *
Disallow:/01/
Disallow:/02/
Disallow:/03/

Prohibit access to a search engine (Badbot in the following example)

User-agent:badbot
Disallow:/

Allow only one search engine access (crawler in the following example)

User-agent:crawler
Disallow:

User: *
Disallow:/

In addition, I think it is necessary to expand the description of the Robots meta Introduction:

The Robots meta tags are focused on specific pages. Like the other meta tags (such as the language used, the description of the page, the keywords, and so on), the robots meta tag is placed in the of the page, specifically to tell the search engine how to crawl the content of the page.

The wording of the Robots meta Tags:

There is no case in the robots meta tag, name= "Robots" means that all search engines can be written as name= "Baiduspider" for a specific search engine. The Content section has four instruction options: index, NOINDEX, follow, nofollow, and the instruction is separated by ",".

The INDEX instruction tells the search robot to crawl the page;

The FOLLOW instruction indicates that the search robot can continue to crawl along the link on the page;

The default value for the Robots Meta tag is index and follow, except for Inktomi, which defaults to Index,nofollow.

In this way, there are four kinds of combinations:

which

It seems that the vast majority of search engine robots comply with the rules of robots.txt, but for the robot meta tags, currently not much support, but is gradually increasing, such as the famous search engine Google fully support, and Google also added a directive " Archive ", you can limit whether Google retains a snapshot of the page. For example:

Represents crawling a page in the site and crawling along the page, but not keeping a snapshot of the page on the Google.

This is my experience today through practice and advice, I hope to help you ~ "Brand Strategy network Www.brhnn.com"