Using a robots file to enhance the page crawl rate

Source: Internet
Author: User

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

I. Understanding the documents of robots

We know that robots.txt file tells spiders what files on the server can be viewed, what is not to be viewed, spiders know these after, you can put all the energy in the pages we allow access to the above, so that the limited weight together.

At the same time, we can not be a little bit the most important point is that robots.txt is the search engine to visit the first file to view the site. With that in mind, we need to do some writing on this one.

Ii. write the Sitemap.xml to the robots file

As mentioned earlier, the robots.txt file is the first file to be viewed by the search engine, so if we will sitemap.xml (site map) File address written robots.txt file inside, will naturally be search engine spiders in the first time to find, but also increased the site page is quickly crawled and included the possibility. This kind of practice, to the new station function is obvious, has the positive and the reality function to the great stations.

Examples of how to do this are:

  

So, whether the new station or old station, in order to make more pages by search engine crawl, it is necessary to write sitemap.xml to robots.txt file.

Iii. acquisition of Sitemap.xml documents

So much has been said before, perhaps many people still do not quite understand, Sitemap.xml document how to capture?

To provide you with a tool, Google is also the official recommendation of the Sitemap.xml online generation tool, http://www.xml-sitemaps.com/, you can choose according to the options for their own site Sitemap.xml generation style. After the sitemap.xml is generated, it is generally necessary to place the file under the root directory.

  

Fully, it is feasible and operable to write the Sitemap.xml address to the robots.txt file to enhance the page crawl rate. Hope that with more friends on the site to promote and optimize the discussion. This article original Taizhou talent network http://tz.qianjiangrc.com. Welcome to reprint, please keep the source.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.