(C #) captures search engine spider and robot

Source: Internet
Author: User

Below is the access log file

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;. Net CLR 2.0.50727;. Net CLR 1.1.4322)


Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1;. Net CLR 2.0.50727;. Net CLR 1.1.4322)


Mozilla/5.0 (compatible; Yahoo! Slurp;


Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; sv1; Maxthon; qqdownload 1.7;. Net CLR 1.1.4322;. Net CLR 2.0.50727;. Net CLR 3.0.04506.648;. Net CLR 3.5.21022)


Mozilla/5.0 (compatible; Yahoo! Slurp;

Open this file, which is used to record normal visitors or robot spider accessing the search bar, extracted from the spider: Mozilla/5.0 (compatible; googlebot/2.1; +

Gigabot/3.0 (

Mozilla/5.0 (compatible; Yahoo! Slurp;

Mozilla/5.0 (compatible; yodaobot/1.0; /;)

Sogou web spider/4.0 (+

Msnbot/1.1 (+

Baiduspider + (+

CodeIt is also very simple:Copy codeThe Code is as follows: string agent = system. Web. httpcontext. Current. Request. servervariables ["http_user_agent"];
If (agent = NULL)
Response. Write ("other ");
Response. Write (agent );

You can try this code and find that the output is similar to: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; tencenttraveler 4.0;. Net CLR 2.0.50727;. Net CLR 1.1.4322)
With this, you can write a spider access record. How can you determine whether a normal user or spider is used? You can roughly determine this: by judging the user's access Source Page, such as from the to show.asp? Id = 11 so we know its source page is, which means he is a normal customer. The source of the spider is not. Record this part and choose from it. It can be recorded in conjunction with the database.

Related Article

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.