Hyper-IP Access/robots.txt solution

Hyper-IP Access/robots.txt
Apache logs are all such records, all access to the/robots.txt, and more than 10 minutes there are tens of thousands of lines. The server CPU often runs to 100%.
Spiders don't crawl robots.txt so often ... The Great God help analysis

66.249.65.147--[17/oct/2014:10:14:38 +0100] "Get/robots.txt http/1.1" 200-
(-it's because I created a blank robots.txt in the root directory of the Web site.) ）

------to solve the idea----------------------
Is it strange that you are sure that CPU is accounted for 100% of this problem?
Shield the robots.txt from the look.
------to solve the idea----------------------
Robots.txt this thing as long as you are careful not to put on the programming, put on is a burden, analysis of his disadvantages, welcome to shoot Bricks:
1, if you do not have the background of the address in the foreground page up to use, spiders will not crawl, for css,js this kind of thing, spiders are not willing to climb.
2, a lot of beginners in the setting robots.txt is very good and obedient, sensitive address put up to shield spider crawl, in fact, as long as strict, spiders can not climb, but the ulterior motives will first look at robots.txt to find the management entrance address.
3, no matter which spider runs over the first crawl is Robot.txt, occupy a little resources
Your situation should not be robots.txt reason, this document should be sent directly, not to 100% of the situation

