PHP code bans searching for the engine spider's real robots.txt is not a hundred percent that can prevent spider crawlers from crawling your website. I have written a small piece of code in combination with some materials, which seems to be able to completely solve this problem. if not, please give me more advice: PHPcodeif (preg_match (& quot; (Googlebot | Msnbot | YodaoBot | Sosospider | baiduspider | google | bai php code disables search engine spider
In fact, Robots.txt is not enough to prevent the spider from crawling your website. I wrote a small piece of code based on some information, which seems to be able to completely solve this problem. if this is not the case, please kindly advise:
PHP code
if(preg_match("/(Googlebot|Msnbot|YodaoBot|Sosospider|baiduspider|google|baidu|yahoo|sogou|bing|coodir|soso|youdao|zhongsou|slurp|ia_archiver|scooter|spider|webcrawler|OutfoxBot)/i", $_SERVER['HTTP_USER_AGENT'])){ header('HTTP/1.1 403 Forbidden'); exit;}
------ Solution --------------------
I understand the concept of LZ, but does each file contain this text or the homepage?
------ Solution --------------------
This trick
------ Solution --------------------
I learned, extinct, too.
------ Solution --------------------
Good !!!
But why? How many crawlers crawl is not more conducive to search engines indexing your site, but more conducive to promotion?
------ Solution --------------------
Discussion
Good !!!
But why? How many crawlers crawl is not more conducive to search engines indexing your site, but more conducive to promotion?
------ Solution --------------------
Discussion
Some items do not need to be indexed by search engines, such as email addresses, phone numbers, names, and other information.
------ Solution --------------------
This method can block "Friendly Spider". of course, these "Friendly Spider" may not fully respect the instructions of robots.txt, but it tells you at least "I'm a ** Spider ".
If it is an unfriendly Spider, this judgment will become ineffective.
