Rookie Seoer talking about robots.txt documents

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Do SEO should know, robots.txt file is a website essential documents, but pony see in many small and medium Enterprises website or no this document, today we will study the importance of robots.txt documents.

One: From the angle of the search engine

When some files on our website don't want to be crawled by search engine spiders, We need to write a robots.txt file, in order to tell spiders, these files can not be crawled, but some companies think that their site does not want to crawl the content of spiders, so they ignored the robots.txt file, then, this is wrong.

As the search engine becomes more intelligent, it visits a new site, Will first visit this website robots.txt file, this also becomes the search engine spider to visit the website a custom, this time, if your website does not robots.txt this file, may cause the search engine spider to your website page not to be able to crawl normally. So pony suggests, even if your site content all allow search engine spiders crawl, you should also build a robots.txt file, which represents the search engine spider friendly.

Second: often check robots.txt documents

Every time we do the maintenance of the website, we are always around the site content and the site outside the chain, whenever our site problems, we are the first of these places to conduct one by one of the investigation, and for the robots.txt file, it is estimated that in addition to the original site was written, has been a long time did not speak, this is wrong.

When a hacker hacked into our site, it does not need to hang on our website black chain, link yellow gambling site, also do not need to add to our content page to its site hidden links, It only needs to modify our small robots.txt file, we can take care of us, and often see some people in search engines to query their own site included, found that there was an anomaly, and then began to check, but the chain and the chain there is no problem, this time, the rush is really hard. This is ignoring the simplest and most important factor----robots.txt files.

Three: Write robots.txt documents to be comprehensive

Robots.txt document writing is also required to standardize, otherwise, like a fake, such as our site on the page when the dynamic address, we know that these dynamic address search engine spiders can not be very good to crawl, so we note in the robots.txt file, some seoer in the second line when the note will be written: Disal Low:/*?*, this is not comprehensive.

We all know that the end of the dynamic Web site are a lot of parameters, such as?,%, etc., so when we do not want search engine spiders crawl Dynamic Web Site page, we will be in the robots.txt file to indicate their own site dynamic URL appears all the parameters. Otherwise, search engine spider still can crawl our website dynamic website, our robots.txt file is just like a dummy.

Article from SEO Tips blog http://www.waszj.com/Village Pony Original, reproduced please specify the source and author