Asp.net (C #) captures search engine spider and robot

Below is the access log file
14:43:22

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;. Net CLR 2.0.50727;. Net CLR 1.1.4322)

14:43:27

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;. Net CLR 2.0.50727;. Net CLR 1.1.4322)

14:44:18

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

14:44:26

Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; sv1; Maxthon; qqdownload 1.7;. Net CLR 1.1.4322;. Net CLR 2.0.50727;. Net CLR 3.0.04506.648;. Net CLR 3.5.21022)

14:45:17

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Open this file, which is used to record normal visitors or robot spider accessing the search bar, extracted from the spider: Mozilla/5.0 (compatible; googlebot/2.1; + http://www.google.com/bot.html)

Gigabot/3.0 (http://www.gigablast.com/spider.html)

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Mozilla/5.0 (compatible; yodaobot/1.0; http://www.yodao.com/help/webmaster/spider /;)

Sogou web spider/4.0 (+ http://www.sogou.com/docs/help/webmasters.htm#07)

Msnbot/1.1 (+ http://search.msn.com/msnbot.htm)

Baiduspider + (+ http://www.baidu.com/search/spider.htm)

CodeIt is also very simple:Copy codeThe Code is as follows: string agent = system. Web. httpcontext. Current. Request. servervariables ["http_user_agent"];
If (agent = NULL)
{
Response. Write ("other ");
}
Else
{
Response. Write (agent );
}

You can try this code and find that the output is similar to: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; tencenttraveler 4.0;. Net CLR 2.0.50727;. Net CLR 1.1.4322)
With this, you can write a spider access record. How can you determine whether a normal user or spider is used? You can roughly determine this: by judging the user's access Source Page, such as from the jb51.net/index.htmfile to show.asp? Id = 11 so we know its source page is jb51.net/index.htm, which means he is a normal customer. The source of the spider is not. Record this part and choose from it. It can be recorded in conjunction with the database.