PHP code summary for determining whether a visitor is a search engine spider or a common user

This article mainly introduces PHP code summary for determining whether a visitor is a search engine spider or a common user. There are always a variety of methods suitable for you to prevent search engine spider from dragging the search engine to death.

This article mainly introduces PHP code summary for determining whether a visitor is a search engine spider or a common user. There are always a variety of methods suitable for you to prevent search engine spider from dragging the search engine to death.

1. recommended method: php judges whether the search engine crawlers are manually accessing the Code, from Discuz x3.2

<? Phpfunction checkrobot ($ useragent = '') {static $ kw_spiders = array ('bot ', 'crawl', 'spider ', 'slup', 'sohu-search ', 'lycos ', 'robozilla'); static $ kw_browsers = array ('msie ', 'netscape', 'Opera ', 'konqueror', 'mozilla '); $ useragent = strtolower (empty ($ useragent )? $ _ SERVER ['HTTP _ USER_AGENT ']: $ useragent); if (strpos ($ useragent, 'HTTP: //') ==== false & dstrpos ($ useragent, $ kw_browsers) return false; if (dstrpos ($ useragent, $ kw_spiders) return true; return false;} function dstrpos ($ string, $ arr, $ returnvalue = false) {if (empty ($ string) return false; foreach (array) $ arr as $ v) {if (strpos ($ string, $ v )! = False) {$ return = $ returnvalue? $ V: true; return $ return ;}} return false ;}if (checkrobot () {echo 'bot crawler' ;}else {echo 'Personal' ;}?>

In actual applications, you can determine that the operation is not performed by a search engine.

<? Phpif (! Checkrobot () {// do something}?>

2. Method 2:

Use PHP to collect spider access logs

$ Useragent = addslashes (strtolower ($ _ SERVER ['HTTP _ USER_AGENT ']); if (strpos ($ useragent, 'googlebot ')! = False) {$ bot = 'Google ';} elseif (strpos ($ useragent, 'mediapartners-Google ')! = False) {$ bot = 'Google Adsense ';} elseif (strpos ($ useragent, 'baidider ider ')! = False) {$ bot = 'baidu';} elseif (strpos ($ useragent, 'sogou spider ')! = False) {$ bot = 'sogou';} elseif (strpos ($ useragent, 'sogou web ')! = False) {$ bot = 'sogou web';} elseif (strpos ($ useragent, 'sosospider ')! = False) {$ bot = 'soso';} elseif (strpos ($ useragent, '360spider ')! = False) {$ bot = '360spider ';} elseif (strpos ($ useragent, 'yahoo ')! = False) {$ bot = 'yahoo ';} elseif (strpos ($ useragent, 'msn ')! = False) {$ bot = 'msn ';} elseif (strpos ($ useragent, 'msnbot ')! = False) {$ bot = 'msnbot ';} elseif (strpos ($ useragent, 'sohu ')! = False) {$ bot = 'sohu ';} elseif (strpos ($ useragent, 'yodaobot ')! = False) {$ bot = 'yodao ';} elseif (strpos ($ useragent, 'twiceler ')! = False) {$ bot = 'twiceler';} elseif (strpos ($ useragent, 'ia _ archiver ')! = False) {$ bot = 'alexa _ ';} elseif (strpos ($ useragent, 'iaarchiver ')! = False) {$ bot = 'Alexa';} elseif (strpos ($ useragent, 'slurp ')! = False) {$ bot = 'yahoo ';} elseif (strpos ($ useragent, 'bot ')! = False) {$ bot = 'other spider ';} if (isset ($ bot) {$ fp = @fopen('bot.txt', 'A'); fwrite ($ fp, date ('Y-m-d H: I: s '). "\ t ". $ _ SERVER ["REMOTE_ADDR"]. "\ t ". $ bot. "\ t ". 'http ://'. $ _ SERVER ['server _ name']. $ _ SERVER ["REQUEST_URI"]. "\ r \ n"); fclose ($ fp );}

Method 3:

We can use HTTP_USER_AGENT to determine whether it is a spider. Search engine spider has its own unique logo. The following column takes a part of it.

Function is_crawler () {$ userAgent = strtolower ($ _ SERVER ['HTTP _ USER_AGENT ']); $ spiders = array ('bot bot', // Google crawler 'baidider ', // Baidu crawlers 'yahoo! Slurp ', // Yahoo crawler 'yodaobot', // youdao crawler 'msnbot '// Bing crawler // more crawler keywords); foreach ($ spiders as $ spider) {$ spider = strtolower ($ spider); if (strpos ($ userAgent, $ spider )! = False) {return true;} return false ;}

The following php code comes with more spider identifiers

Function isCrawler () {echo $ agent = strtolower ($ _ SERVER ['HTTP _ USER_AGENT ']); if (! Empty ($ agent) {$ spiderSite = array ("TencentTraveler", "Baiduspider +", "BaiduGame", "Googlebot", "msnbot", "sososospider + ", "Sogou web spider", "ia_archiver", "Yahoo! Slurp "," YoudaoBot "," Yahoo Slurp "," MSNBot "," Java (Often spam bot) "," BaiDuSpider "," Voila "," Yandex bot ", "BSpider", "twiceler", "Sogou Spider", "Speedy Spider", "Google AdSense", "Heritrix", "Python-urllib", "Alexa (IA Archiver) "," Ask "," Exabot "," Custo "," OutfoxBot/YodaoBot "," yacy "," SurveyBot "," legs "," lwp-trivial ", "Nutch", "StackRambler", "The web archive (IA Archiver)", "Perl tool", "MJ12bot", "Netcraft", "MSIECraw Ler "," WGet tools "," larbin "," Fish search ",); foreach ($ spiderSite as $ val) {$ str = strtolower ($ val ); if (strpos ($ agent, $ str )! = False) {return true ;}} else {return false ;}} if (isCrawler () {echo "Hello spider! ";} Else {echo" you are not a spider! ";}

Method 4:

<? Php $ flag = false; $ tmp = $ _ SERVER ['HTTP _ USER_AGENT ']; if (strpos ($ tmp, 'googlebot ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'baidider ider ')> 0) {$ flag = true ;} else if (strpos ($ tmp, 'yahoo! Slurp ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'msnbot ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'sosospider ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'yodaobot ')! = False | strpos ($ tmp, 'outfoxbot ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'sogou web spider ')! = False | strpos ($ tmp, 'sogou Orion spider ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'fast-webcrawler ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'gaisbot ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'ia _ archiver ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'altavista ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'lycos _ spider ')! = False) {$ flag = true;} else if (strpos ($ tmp, 'inktomi slurp ')! = False) {$ flag = true;} if ($ flag = false) {header ("Location :". $ _ SERVER ['request _ URI ']); // automatically go to the corresponding webpage // $ _ SERVER ['request _ URI '] as the path behind the domain name // or change to header ("Location :"); exit () ;}?>