Teach Novice How to create Robots.txt file, in front of the master to swim, sin!

Source: Internet
Author: User

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

How do I create a robots.txt file?
You can create this file in any text editor. The file should be an ASCII-encoded text file, not an HTML file. File names should use lowercase letters.

The simplest robots.txt files use two rules:
(1) User: Bots that apply to the following rules
(2) Disallow: pages to block
These two lines are considered an entry in the file. You can include as many entries as you want. You can include multiple Disallow rows and multiple
user in an entry. What should

be listed in the user line? The
user is a specific search engine rover. The Network rover database lists a number of common bots. You can set the entry
(by column name) or set to apply to all bots (by listing asterisks) that apply to a particular rover. Entries applied to all bots should resemble the following entries:

user-agent:*
Google uses a variety of different bots (user agents). The rover for 11510.html "> network search is Googlebot. Other bots, such as Googlebot-mobile and googlebot-
Image, follow the rules that you set for Googlebot, and you can also set additional rules for those specific bots. What should

be listed in the Disallow line? The
Disallow the page you want to intercept. You can list specific URLs or URL patterns. Entries should begin with a forward slash (/).

to block the entire Web site, use a forward tilt.

disallow:/
to block the directory and all content in it, add a forward tilt after the directory name.

disallow:/private_directory/
to block Web pages, list the page. The

disallow:/private_file.html
URLs are case-sensitive. For example, disallow:/index.htm blocks http://www.gz-kongtiao.cn/index.htm, but does not
intercept http://www.gz-kongtiao.cn/index.htm.

    above is my view of the relevant information summed up, I hope to be useful!

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.