Spiders crawl Web sites and crawl pages that don't exist.

Source: Internet
Author: User

650) this.width=650; "src=" Http://183.61.143.148/group1/M00/01/FD/tz2PlFQJWkqC7EEGAAA09Xmt0bo668.jpg "style=" border:0px; "/>

seoer You need to check the Web site's server log regularly to get a handle on where and what page the spider crawled from our site. But sometimes it is found that spiders crawl a few pages that our site does not exist, today SEO tutorial

1 , how spiders find links to our website ?

We all know that the spider is crawling along the link crawl, he will automatically extract all the links in the site, save the storage, and then crawl, which is why we feel that the site contains less or just released articles are not included, we will say "primer spider", in fact, the spider is hair outside the chain, When the link is discovered, the spider will crawl and then, after a series of complex algorithms, decide whether to release the page.

2 , why do you crawl to a page that doesn't exist ? ?

In general, there are so few questions

a , by the opponent malicious brush outside the chain, that is, the competitor malicious to you sent a lot of non-existent page links, when the spider found to crawl

b , the outside chain, and later because of the site revision and some links did not remove timely, spiders regularly visit the time to crawl the link

C , this type of old domain name only, the previous site structure and our current site structure is inconsistent, also similar to b in the argument

D , Robots The file doesn't restrict spiders .

e , the cause of the program causes the search engine spider to fall into the black hole

F , URL Submit or Ping to Baidu address is wrong

Spiders crawl does not exist in the page, in general, because of these problems, the most important reason is the chain part, so we have to regularly check the external chain of our website.

these aspects, generally speaking, we only need to do this, the basic can reduce the occurrence of this situation, if it is caused by the link, then go to the Baidu Webmaster platform use outside the chain reject tool rejected, and submit the site of the dead link, while using Robots Block Spiders crawl these contents, and if it is a program problem, fix the program.

If you want to know more about the relevant knowledge and content, e Mentor Network for you to recommend the appropriate SEO advanced Tutorials .


Spiders crawl Web sites and crawl pages that don't exist.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.