PHP code prohibits search engine spiders
In fact, robots.txt is not a total can stop spiders crawling your site, I combined some data to write a small piece of code, seemingly can completely solve the problem, the wrong place please give us a lot of advice:
PHP Code
if (Preg_match ("/(googlebot| Msnbot| Yodaobot| sosospider|baiduspider|google|baidu|yahoo|sogou|bing|coodir|soso|youdao|zhongsou|slurp|ia_archiver|scooter| spider|webcrawler| Outfoxbot)/i ", $_server[' Http_user_agent ')) { header (' http/1.1 403 Forbidden '); Exit;}
------Solution--------------------
LZ thought I got it. But does each file contain this text or is the homepage included?
------Solution--------------------
It's a trick.
------Solution--------------------
Learned, the extermination teacher too
------Solution--------------------
Not bad!!!
But why stop it? Spiders are not more crawling is not more conducive to search engines to include your site, more conducive to the promotion?
------Solution--------------------
discuss
Not bad!!!
But why stop it? Spiders are not more crawling is not more conducive to search engines to include your site, more conducive to the promotion?
------Solution--------------------
discuss
Some things do not want to be included in the search engine, such as related to e-mail address, telephone number, name and other information
------Solution--------------------
This way can stop the "friendly spider", of course, perhaps these "friendly spider" does not fully respect the instructions of robots.txt, but it tells you at least "I am xx spider."
If the spider is unfriendly, this judgment will be ineffective.
