A brief discussion on the error of three points in the robots file that we can easily appear

The Robots.txt file seems to have only a few lines of letters, but in fact there are many details that need our attention. Because if you do not pay attention to these details, some statements will not be effective, or even have a hair effect. and Robots.txtt file is the search engine into our site after the first access to the file, its written good is related to the site's SEO can be carried out smoothly. The following is an example of three errors in the details that are easy to appear when robots.txt files.

　Error one: Allow and disallow in reverse order

First, let's look at a robots.txt statement:

User-agent: *

Allow:/

Disallow:/abcd/

At the beginning of this statement we can get the purpose of telling the search engine that the page below the ABCD directory cannot crawl, but all other pages can. But what about the actual effect of this statement? The effect is the opposite, why? We know that the search engine spider's execution of the rules in the robots file is from top to bottom. This will cause disallow statement invalidation, the correct way is to put the disallow:/abcd/placed allow:/Before we can play the desired effect.

　Error two: When a page is blocked, the page name is not with a slash "/"

I think that a lot of webmaster is also easy to ignore, for example, we want to screen the search engine in the root directory of the abcd.html page, some people on the robots may write: Disallow: Abcd.html, on the surface may be no problem, but the author would like to ask you first to tell search engine shielding this page under what directory? If we don't take it, the search engine spider can't identify which page it is on. The correct way to do this is: disallow:/abcd.html, so that you can really mask the abcd.html page located underneath the root directory.

error Three: The back of the screen did not follow the slash "/"

For example, if we want to block the page below the/abcd/directory, some people may write disallow:/ABCD. Is there a problem with this writing? There are still problems with this writing, and there is a big problem. This will mask all the pages below the/abcd/directory. But this will also be passed to the search engine, which is to screen all the pages that start with/ABCD. The effect equals disallow:/abcd ' *. This error will make a big impact on the site. The right thing to do is to remember to add a slash "/" Behind the directory you want to mask.

For the importance of robots.txt file must be a lot of webmaster have experience. If we do not pay attention to these small details of writing will not only lose the original effect of robotss.txt, but also can cause unnecessary reverse effect. I hope the three details presented in this article will help you when you write the document. Article by the Korean film http://www.hgdyy.com/Webmaster original, reprint please keep our address.