Analysis: Why spiders do not crawl my site

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Just on seowhy see a very interesting question, ask the following questions: (solution) Why my station, again no Baidu Spider came. From the above problems can be seen, the webmaster encountered such a problem must be very egg pain. Their carefully done site, did not think spiders did not climb over. So what causes spiders not to crawl? Let me analyze it.

Encounter such a problem, the situation only two kinds: one, you shield spider crawling. You're shielded by spiders.

One, you shielded the spider.

We shielded the spiders, and of course spiders could not crawl our site to pull.

1. First of all, we want to test their own web site robots.txt file, to see whether the file is written correctly, whether there is a case of confusion, whether the initial stage of the station has shielded all spiders crawling instructions forget to modify. Under normal circumstances robots.txt detection of things directly to Webmaster tools check it

2. Whether the server has shielded spiders from crawling. Do not think this is impossible, the original mainframe house, Baidu Spider as a hacker attack to shield, do not know how many people were K station. If it is the host to screen the spider's crawling as soon as possible change the host, time can't afford.

You're shielded by spiders.

1. View their own domain name before the owner to see whether the domain name before the illegal site. If it is then the spider does not crawl is completely justified, like Maas before the forum that domain name before the Liuhecai station. So far Sogou and Google spiders have never been.

2. View the site of their own with the server, whether there is a very serious violation of cheating, if any, search engines are likely to block the entire IP site, for their own web site development health or select the independent IP server bar.

If you find the crawling log of your site did not appear spider visit, then we should consider their own site whether the above situation, should be timely clean-up. Then submit to the relevant search engine, or use high-quality outside the chain to guide the spider crawling. (Source: MAS-micro-Internet http://www.ma4.net Reprint annotated origin)