The new station on-line 12 hours to be included analysis spider creeping law

Source: Internet
Author: User
Keywords Network promotion SEO

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

This morning, the use of site:www.wengerbag.com search, found Granville bag net was Soso included, 12 hours from the line. Soso will always be that kind of understanding! Online 15 hours later, Google also included a Granville bag network. Google to the new station really good enough friendly!

  

  

In fact, September 8 night 8 o'clock more than the Wai-wai Bag online after the chain, from the log to see Google robot is a few minutes after 8:28 to patronize the site, Baidu Spider appeared after half an hour, Soso Spider is late 20 minutes, that is, 50 minutes later than Google robot.

From the spider crawling Granville Bag network situation, Soso is very strange, the first is to catch robots.txt, then there is no next--is because the night too busy? Until one o'clock in the morning, Soso and grabbed the home and the necessary script, and then lurk up. In the morning 8, Soso again to crawl a bit home. This period Granville bag Net has not updated, so the homepage did not change, Soso also retreated again.

Google robot is also first crawl robots.txt files, and then immediately grab the home page. 1.5 hours later spend 9:52, Google robot again appear or crawl robots.txt culture, and then crawl home. Another one hours to 11:04, Google Robot began to explode, it is still first grab the robots.txt files, and then grab the directory page and other articles page, the more strange is that these crawl is not done immediately, between the time interval of one hours. The crawl of all pages is completed before one o'clock in the morning. As of 11 o'clock in the morning press, Google robot did not come to Granville bag net.

Baidu spiders crawl characteristics are very obvious. From 8th 8:30 P.M. to 9th 11 o'clock in the morning, Baidu Spider to patronize Granville Bag Network 11 times. In the trial of 11 times, Baidu Spider's Temptation law is very interesting, 8:56 crawl home, half a minute to crawl home again, another three minutes to come again, another three minutes to come again, it is estimated that the home page did not change, and after half an hour to come, and then after a quarter of an hour, and then another one or a quarter of a clock to come, 10:48 Baidu Spider full Retreat. Today 9 o'clock in the morning, Baidu Spider in two minutes two times to patronize Granville Bag net home, probably see the home page or not update, nearly 11 point when again come crawling again.

Baidu Spider to Granville bag net every time is straight to the home page, on the robots.txt ignored, that Baidu spider or a bit of violence-your statement I do not care! At the same time for a new contact station, Baidu only focus on the home page, the internal pages indifferent.

From the crawl law of three spiders, stationmaster should update the site in time, especially the homepage, must keep the update status every day. For the new station, if you want to quickly arouse the interest of spiders, you can consider the practice of "eating less and more meals", every once in a while to send an article, this effect than a one-time issue all the effect is good.

Related Article

Beyond APAC's No.1 Cloud

19.6% IaaS Market Share in Asia Pacific - Gartner IT Service report, 2018

Learn more >

Apsara Conference 2019

The Rise of Data Intelligence, September 25th - 27th, Hangzhou, China

Learn more >

Alibaba Cloud Free Trial

Learn and experience the power of Alibaba Cloud with a free trial worth $300-1200 USD

Learn more >

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.