Wu Xianmao: The application of robots.txt and Sitemap

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

First I explain what is Robots.txt,robots.txt is the search engine to visit the site to see the first file. The Robots.txt file tells the spider what files can be viewed on the server. When a search spider accesses a site, it first checks to see if there is a robots.txt in the root directory of the site, and if so, the search robot will determine the scope of the access according to the contents of the file; If the file does not exist, all search spiders will be able to access all pages that are not password protected on the site.

However, many people know that there are robots, but do not know how to do it, today I say how to write robots.txt first of all you have to think about those Web site directory is not to be searched spiders to crawl, which is needed to be crawled, this is very important, If you write a robots may be blocked site to the search engine included.

robots.txt files must be written on the root of the site, for example:

Www.***com/robots.txt

Not like this.

Www.***.com/seo/robots.txt

Remember to be in the root directory right, and then there are two writing

User: Bots that apply the following rules

Disallow: Web pages to intercept

User-agent:googlebot (Baiduspider) This is Google (Baidu) spider

Disallow:/seo

This means that I do not want Google or Baidu spiders to crawl my SEO this folder inside the things, * the words on behalf of all. If you build an empty or do not build a robots, the site will be included in all the content of the search engine, and then many people will ask, the site is not included the more the better?

I would like to make an analogy, your site generated static, but the previous dynamic is still in the spider is to crawl both included, and then the search engine will think that two titles of the same article appeared on the same site will think you are cheating, there may be down right.

Sitemap (Site map)

Mainly because many sites have no other links to the content, in order to link these links better, so that spiders can crawl more resources. If it is Google, you can register an administrator tool, generated by the XML file submitted to Google, Baidu without administrator tools can be written in the robots.txt inside the Sitemap

The wording is as follows:

Sitemap:http://www.***.com/sitemap.xml

Sitemap:http://www.***.com/sitemap.txt

Sitemap:http://www.***.com/sitemap.html

These three forms can, generally automatically generate XML, if not generate Sitemap can go online to find tools or use some open source CMS will have their own! This article by the www.ainatec.com webmaster feeds!