SEO Summary: The details of the design of robots are included in which places (i)

Website optimization is a systematic project, there are many things must be experimental research and summary. For some of the large-scale web site, some simple SEO details can have an impact on the overall, and sometimes will directly affect the site's inclusion, transformation and even profitability. So for the scale of the site, some SEO details are compelled to mention, have to pay attention to. Today and you sum up a few knowledge about robots, hope to let everyone benefit.

The use of a lot of webmaster files are familiar, but in detail may not care, may not be proficient. It contains a lot of knowledge points search engines do not give a detailed description or use methods. After a lot of summary and see the data on the basis of this article and you summarize the operation of the document should pay attention to the details.

Number of first, user-agent

General Web site robots.txt files do not appear in a number of user-agent records, generally is a suitable, but if one does not appear, it is allowed.

Second, spider spider case

Search engine spiders will ignore the case when looking at the robots, but for insurance, there are two different Baidu spiders, such as Taobao.

Third, allow and disallow order

The sequence of problems will lead to crawl errors, if the allow placed behind, sometimes will appear to be included in the content is not included, examples:

If you want a search engine to include the only b.html under a folder, then write this:

Allow:/a/b.html

disallow:/a/

If the sequence is written in reverse, after the first match succeeds, b.html will not be included.

Four, remember the back slash

6298

The backslash means the root directory, so disallow:/the meaning of the ghost is to prohibit the inclusion of all pages and folder content. This backslash is important, and a backslash is allowed to include all the pages in the search engine.

V, #表示注释

In Robots.txt, #开头均被视为注释, this is the same as the practice in UNIX. Don't accidentally write wildcard characters wrong.

Six, do not appear space

Some robots file in front will not pay attention to a space, such a problem should try to avoid, or some robot will not understand meaning.

Well, space is limited, this article will first write these six aspects, the following is also a directory case, meta cumulative value, the adoption of the most restrictive parameters, such as the introduction, are very useful details, Robots.txt Although the main direction of optimization, but it does need to pay attention to a direction, if there is a problem, the entire site will be included in the impact. We should not drift with the flow, we will write how others write. But to understand the meaning of it, write a document that matches your website.

This article from the left 360 Slimming coffee (http://www.kfzxrj.com), reproduced please keep the link, thank you!