How to stop unfriendly search engine spider crawler?

Found on the server today, MySQL traffic is very high. Then look at the log, found an unfriendly spider crawler, looked at the time a second visit page 7, 8 times, and access to the site of the entire station collection page. is to not listen to the query database.
I would like to ask you how to avoid this kind of problem. Now I'm still in this IP address

Reply to discussion (solution)

There's no good way. You can observe which IP access is frequent, and what is suspected is forbidden.

program, you can set the time interval for the same IP two access.

There's no good way. You can observe which IP access is frequent, and what is suspected is forbidden.

program, you can set the time interval for the same IP two access.
Thank you, just looked at Discuz, as if there is a robots.txt can set the crawler access site, this is useful?

Found on the server today, MySQL traffic is very high. Then look at the log, found an unfriendly spider crawler, looked at the time a second visit page 7, 8 times, and access to the site of the entire station collection page. is to not listen to the query database.
I would like to ask you how to avoid this kind of problem. Now I'm still in this IP address
Learn it first!

Robots.txt are generic, search engines follow the specifications, they will crawl your robots.txt first, according to the constraints of the crawl.

Key you look at the access log to see who the reptile looked at, they will be on the user-agent head to mark what the Web crawler.

First configure the format of Access_log print out user-agent, restart the server, wait a while to see Access_log, see if it is normal crawler, not normal crawler you can only restrict IP access.

Baidu's various spider names:

Product name corresponds to User-agent
Web Search Baiduspider
Wireless Search Baiduspider-mobile
Image Search Baiduspider-image
Video Search Baiduspider-video
News search Baiduspider-news
Baidu Search Tibetan Baiduspider-favo
Baidu Alliance Baiduspider-cpro

This is the crawler of Baidu.

Yes, but now I'm going to write a program that controls the interval between each IP query.

Robots.txt to most of the crawler still useful drops, directly block off, but for some crawlers, such as Baidu, is unstoppable.
Suggested Plus.

Originally robots.txt can be, but there are always some people do not obey the rules. So keep logging, then block out some IP addresses ...

Of course, you can also use HTTPS ... Oh

Since this reptile is 1 seconds 7.8 times, do you think he will look at your robots.txt?
Baidu does not look, don't say this kind of collector.

Judge the frequency or analyze its UA head, and resolutely forbid it. Don't mention it.

Collector Crawler A principle, right?

