How to reduce the request of Yahoo search to the website

Intermediary transaction http://www.aliyun.com/zixun/aggregation/6858.html ">seo diagnose Taobao guest cloud host technology Hall

Due to the 6184.html "> Yahoo search for a large number of network crawl systems, therefore, our Web server will be from different yst (Yahoo search technology) crawler client IP address login request. Different crawler systems work together to limit any activity from a single network server. The so-called single network server, is judged by the IP address. Therefore, if our server host has more than one IP, its activities will be at a higher level.

Yst has a specific extension, which can be applied in the robots.txt of our server root directory, which allows you to set a low crawl request frequency for Yahoo search crawlers. You can add crawl-delay:xx instructions in robots.txt, where "xx" refers to the minimum delay in seconds when the crawler program enters the site two times. If the crawler frequency is a burden on our server, we can set this delay to any appropriate number, such as 50 or 200.

For example: We want to set a 5-second delay, you can add the following statement in Robots.txt:

User-agent:slurp

Crawl-delay:5

can also set longer time delay, in this network wisdom to no longer repeat.