Using a robots file to enhance the page crawl rate

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

I. Understanding the documents of robots

We know that robots.txt file tells spiders what files on the server can be viewed, what is not to be viewed, spiders know these after, you can put all the energy in the pages we allow access to the above, so that the limited weight together.

At the same time, we can not be a little bit the most important point is that robots.txt is the search engine to visit the first file to view the site. With that in mind, we need to do some writing on this one.

Ii. write the Sitemap.xml to the robots file

As mentioned earlier, the robots.txt file is the first file to be viewed by the search engine, so if we will sitemap.xml (site map) File address written robots.txt file inside, will naturally be search engine spiders in the first time to find, but also increased the site page is quickly crawled and included the possibility. This kind of practice, to the new station function is obvious, has the positive and the reality function to the great stations.

Examples of how to do this are:

So, whether the new station or old station, in order to make more pages by search engine crawl, it is necessary to write sitemap.xml to robots.txt file.

Iii. acquisition of Sitemap.xml documents

So much has been said before, perhaps many people still do not quite understand, Sitemap.xml document how to capture?

To provide you with a tool, Google is also the official recommendation of the Sitemap.xml online generation tool, http://www.xml-sitemaps.com/, you can choose according to the options for their own site Sitemap.xml generation style. After the sitemap.xml is generated, it is generally necessary to place the file under the root directory.

Fully, it is feasible and operable to write the Sitemap.xml address to the robots.txt file to enhance the page crawl rate. Hope that with more friends on the site to promote and optimize the discussion. This article original Taizhou talent network http://tz.qianjiangrc.com. Welcome to reprint, please keep the source.