On indexing of robots.txt files by search engines

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Today in the search site included in the time, saw a never seen in the collection of a Web site, the following figure:

Because it has never been included in the robots.txt such a Web site, so I looked up the data, search to Google has already told us which types of files will be indexed, but I did not encounter not aware of it! Here I quote a Google article for your reference, the text is as follows:

What types of files can Google index?

Google can index most types of Web pages and file content. Our most frequently indexed file types include:

Adobe Flash (. swf)

Adobe Portable Document Format (. pdf)

Adobe Postscript (. PS)

Autodesk Design Web Format (. DWF)

Google Earth (. KML,. kmz)

GPS Interchange Format (. GPX)

Hancom Hanword (. hwp)

HTML (. htm,. html, other file name extensions)

Microsoft Excel (. xls,. xlsx)

Microsoft PowerPoint (. ppt,. pptx)

Microsoft Word (. doc,. docx)

OpenOffice presentation (. ODP)

OPENOFFICE spreadsheet (. ods)

OpenOffice text file (. odt)

Rich Text Format (. rtf,. wri)

Scalable Vector Graphics (. svg)

Tex/latex (. Tex)

Text files (. txt,. text, other file name extensions), including source code in common programming languages:

Basic source code (. bas)

C + + source code (. C,. CC,. cpp,. cxx,. h,. hpp)

C # source code (. cs)

Java source code (. java)

Perl source code (. pl)

Python source code (. py)

Wireless Markup Language (. WML,. WAP)

XML (. xml)

If you are using the filetype: operator (for example, Filetype:xml Galway) in google search, Google searches for files with an. xml extension, rather than a file of type XML.

Not all URLs or Web pages have file extensions that are. html. Some pages, such as http://www.google.com, do not have an extension at all, and some pages do not have an extension that matches their file type. Searching only for a specific file type returns a Web page with the appropriate file name extension, but may have fewer related results.

Quoted article from: http://support.google.com/webmasters/bin/answer.py?hl=zh-Hans&answer=35287

After reading this help article I try to search to see if it is really like that, we use the lookup file command to search, Filetype:txt, get the following results:

You can see in the first place is a site in the United States area, open look like this:

According to the TXT text file search results, is it possible to contact our robots file is also a text format? Is it going to be indexed? The answer is YES!

Let's find out if robots.txt is actually indexed as we said above. We enter a filetype:txt in the search box, meaning to find the index of the robots TXT file. Search results are as follows:

The search results confirm that I was initially "indexed by search engines to robots.txt files?" This doubt now has a clear answer, that is: Search engine on the robots.txt file will be included!

Today and everyone to share here, if there are different views can be discussed with me ha!