How to Prevent robot from harassing your webpage

I saw an article on www.aspalliance.com this morning. Article ( Stopping automated Web Robots visiting ASP/ASP. NET websites
, Http://aspalliance.com/1018_Stopping_Automated_Web_Robots_Visiting_ASPASPNET_Websites ),
It mainly talks about how to take some measures to prevent the robot from over-capturing your website. After reading this, some things are worth exploring and summarized as follows:

1. Identify some robot Reference Standards
Large numbers of requests from a single IP address or a range of IP addresses within the same subnet (I. e. the first three numbers of the IP address are identical ).

·Large numbers of requests for database driven content compared to the rest of the website.

·Required requests made from browsers that do not support ASP sessions.

·Lots of and increasing numbers of website visitors, but no corresponding increase in transactions (e.g. Sales !).

·Large numbers of spam or automatic requests being generated from online forms.
2. Go to http://www.robotstxt.org/wc/norobots.html. you can find the defense robots' standard proposal file that is provided by an organization and put it under the root directory of the website, for example
User-Agent :*
Disallow :/
Disable all robots

Allow all robot access:

User-Agent :*
Disallow:

User-Agent :*
Disallow:/cyberworld/MAP/do not allow robot to access files in the/cyberworld/map directory

User-Agent: cybermapper allows cybermapper to be a robot.
Disallow:

User-Agent :*
Disallow:/cyberworld/MAP/
Disallow:/tmp/
Disallow:/foo.html does not allow foo.html.

32.16if you do not set robot.txt easily, you can also perform operations in Meta, such as using
<Meta name = "Robots" content = "noindex, nofollow">
You can set a defense robot for a page separately.

4. Slow down the crazy robot access. If you find that robot is crazy to access your site, which may reduce the efficiency
Robot access,
User-Agent: slurp
Crawl-delay: 10

Is for Yahoo, specific to the http://help.yahoo.com/help/us/ysearch/slurp/slurp-03.html
See the details.
But in fact, some robots are intelligent, and sometimes they won't be so stupid to visit.

5. If you find that a page is not only a robot, but a malicious attack, you can find it in the article.

Http://www.peterblum.com/VAM/VISETools.aspx#SDM.
Try this tool

Of course, it seems that cnblogs has been plagued by robot. Therefore, you are welcome to give more comments and put forward your own comments.
Method.