Search engine spider stealth three methods

Search engine indexing is mainly done by search engine spider. Each mainstream search engine has its own spider crawlers, such as googlebot (Google SPIDER) and baiduspider (Baidu spider) yahoo slurp (Yahoo SPIDER), msnbot (msn spider), Scooter (Altavista SPIDER), lycos_spider _ (t-rex), fast-webcrawler (alltheweb spider) and ia_archiver (Alexa spider) slurp (Inktomi spider. You can use the following three methods to create a search engine.

1. rel = nofollow

This kind of unrecognized link search tag can tell the search engine spider to skip without indexing. For details, see "How To Make paid links without being punished by Google", which is effective for Google, Yahoo, and MSN (Live Search) and ineffective for Baidu.

2、robots.txt

You only need to create a robots.txt file under the root directory of your website. Enter disallow: string in the file, for example:

Disallow: newsunday.html indicates that spider crawlers are not required to download the file, which is not included or indexed.

Disallow:/newsunday/indicates that all files in the newsunday folder are not indexed or downloaded.

If you want to see a specific example, you can see what Google robots.txt looks like.

3. Use code to implement

This is the focus of this article, but first it is declared that this method should not be used in terms of search engine cheating, and it cannot be used in violation of Search Engine rules or spoofing search engines. The effect of code implementation is that the user can view the content defined by the Code and display it normally, while the search engine crawlers do not display the content browsed by the user. This article takes Baidu Spider as an example:

ASP

$ Useragent = server. getvariables ("http_user_agent ");
If not instr (lcase ($ useragent), "baiduspider") then
'The content code that the user can see during normal browsing
Else
'Code and link that search engine spider can see
End if

PHP

$ Useragent = $ _ server ['HTTP _ user_agent '];
If (stristr (strtolower ($ useragent), 'baidider ider ') === false ){
// What the user can see during normal browsing
// <! -Your DEMO code->
}
Else {
// The code and link that the search engine spider can see
}

JSP

<%
String useragent = request. getheader ("User-Agent ");
If (useragent. tolowercase (). indexof ("baiduspider ")! =-1 ){
// Content code that the user can see during normal browsing
}
Else {
// The code and link that the search engine spider can see
}
%>

Javascript

If (navigator. useragent. tolowercase (). indexof ("baiduspider") <=-1 ){
// What the user can see during normal browsing
}
Else {
// The code and link that the search engine spider can see
}

After reading this article, you may feel that the code in this article is meaningless. In fact, if you think about it carefully, you can come up with a lot of things. I will not talk about the specifics. If you think it is useful, you can add it to your favorites. If you think it is not very useful, you should be familiar with the common sense of search engines.