Home > Website Builders > Website Operations

List of spider names in the world's major search engines

Source: Internet
Author: User
Developer on Alibaba Coud: Build your first app with APIs, SDKs, and tutorials on the Alibaba Cloud. Read more ＞

Intermediary transaction SEO diagnosis Taobao guest Cloud host technology Hall

This article records the search spiders that need to be set up in the world's more famous Robots.txt list. How to set that directory do not want to be indexed by the search engine can be set.

Of course, it has to be set from Robots.txt.

The following are the more famous search engine spider name:

Google Spider: Googlebot
Baidu Spider: Baiduspider
Yahoo Spider: Yahoo slurp
MSN Spider: Msnbot
AltaVista Spider: Scooter
Lycos Spider: Lycos_spider_ (Rex)
AllTheWeb Spider: fast-webcrawler/
Inktomi Spider: slurp

For reference, refer to this article:
User (user agent Settings): (spider name)
Refusal: (file name)

User-agent:black Hole
Disallow:/

User-agent:titan
Disallow:/

User-agent:webstripper
Disallow:/

User-agent:netmechanic
Disallow:/

User-agent:cherrypicker
Disallow:/

User-agent:emailcollector
Disallow:/

User-agent:emailsiphon
Disallow:/

User-agent:webbandit
Disallow:/

User-agent:emailwolf
Disallow:/

User-agent:extractorpro
Disallow:/

User-agent:copyrightcheck
Disallow:/

User-agent:crescent
Disallow:/

User-agent:nicerspro
Disallow:/

User-agent:wget
Disallow:/

User-agent:sitesnagger
Disallow:/

User-agent:prowebwalker
Disallow:/

User-agent:cheesebot
Disallow:/

User-agent:mozilla/4
Disallow:/

User-agent:mozilla/5
Disallow:/

user-agent:mozilla/4.0 (compatible; MSIE 4.0; Windows NT)
Disallow:/

user-agent:mozilla/4.0 (compatible; MSIE 4.0; Windows 95)
Disallow:/

user-agent:mozilla/4.0 (compatible; MSIE 4.0; Windows 9
Disallow:/

User-agent:ia_archiver
Disallow:/

user-agent:ia_archiver/1.6
Disallow:/

User-agent:alexibot
Disallow:/

User-agent:teleport
Disallow:/

User-agent:teleportpro
Disallow:/

User-agent:wget
Disallow:/

user-agent:miixpc
Disallow:/

User-agent:telesoft
Disallow:/

User-agent:website Quester
Disallow:/

User-agent:webzip
Disallow:/

user-agent:moget/2.1
Disallow:/

user-agent:webzip/4.0
Disallow:/

User-agent:webstripper
Disallow:/

User-agent:websauger
Disallow:/

User-agent:webcopier
Disallow:/

User-agent:netants
Disallow:/

User-agent:mister PiX
Disallow:/

User-agent:webauto
Disallow:/

User-agent:thenomad
Disallow:/

User-agent:www-collector-e
Disallow:/

User-agent:rma
Disallow:/

User-agent:libweb/clshttpdisallow:/

User-agent:asterias
Disallow:/

User-agent:turingos
Disallow:/

User-agent:spanner
Disallow:/

User-agent:infonavirobot
Disallow:/

user-agent:harvest/1.5
Disallow:/

User-agent:extractorpro
Disallow:/

user-agent:bullseye/1.0
Disallow:/

user-agent:mozilla/4.0 (compatible; Bullseye; Windows 95)
Disallow:/

User-agent:crescent Internet ToolPak httpole Control v.1.0
Disallow:/

user-agent:cherrypickerse/1.0
Disallow:/

user-agent:cherrypickerelite/1.0
Disallow:/

user-agent:webbandit/3.50
Disallow:/

User-agent:nicerspro
Disallow:/

User-agent:microsoft URL Control-5.01.4511
Disallow:/

User-agent:dittospyder
Disallow:/

User-agent:foobot
Disallow:/

User-agent:webmasterworldforumbot
Disallow:/

User-agent:spankbot
Disallow:/

User-agent:botalot
Disallow:/

user-agent:lwp-trivial/1.34
Disallow:/

User-agent:lwp-trivial
Disallow:/

User-agent:bunnyslippers
Disallow:/

User-agent:microsoft URL Control-6.00.8169
Disallow:/

User-agent:urly Warning
Disallow:/

User-agent:wget
Disallow:/

user-agent:wget/1.5.3
Disallow:/

User-agent:linkwalker
Disallow:/

User-agent:cosmos
Disallow:/

User-agent:moget
Disallow:/

User-agent:hloader
Disallow:/

User-agent:humanlinks
Disallow:/

User-agent:linkextractorpro
Disallow:/

User-agent:offline Explorer
Disallow:/

Editor: Windear
Starting: SEO, SEM conspiracy, webmaster Resource Information Network
Address: http://www.seoplot.com,/
Reprint please source and keep the connection! Otherwise it will be

User-agent:mata Hari
Disallow:/

User-agent:lexibot
Disallow:/

User-agent:offline Explorer
Disallow:/

User-agent:web Image Collector
Disallow:/

User-agent:the intraformant
Disallow:/

user-agent:true_robot/1.0
Disallow:/

User-agent:true_robot
Disallow:/

user-agent:blowfish/1.0
Disallow:/

User-agent:jennybot
Disallow:/

user-agent:miixpc/4.2
Disallow:/

User-agent:builtbottough
Disallow:/

user-agent:propowerbot/2.14
Disallow:/

user-agent:backdoorbot/1.0
Disallow:/

User-agent:tocrawl/urldispatcher
Disallow:/

User-agent:webenhancer
Disallow:/

User-agent:tighttwatbot
Disallow:/

User-agent:suzuran
Disallow:/

User-agent:vci webviewer VCI webviewer Win32
Disallow:/

User-agent:vci
Disallow:/

user-agent:szukacz/1.4
Disallow:/

User-agent:queryn metasearch
Disallow:/

User-agent:openfind Data Gathere
Disallow:/

User-agent:openfind
Disallow:/

User-agent:xenu ' s Link sleuth 1.1c
Disallow:/

User-agent:xenu ' s
Disallow:/

User-agent:zeus
Disallow:/

User-agent:repomonkey Bait & tackle/v1.01
Disallow:/

User-agent:repomonkey
Disallow:/

User-agent:zeus 32297 Webster Pro V2.9 Win32
Disallow:/

User-agent:webster Pro
Disallow:/

User-agent:erocrawler
Disallow:/

user-agent:linkscan/8.1a Unix Disallow:/

User-agent:kenjin Spider
Disallow:/

User-agent:keyword density/0.9
Disallow:/

User-agent:kenjin Spider
Disallow:/

User-agent:cegbfeieh
Disallow:/

Different:

User-agent:larbin
Disallow:/

user-agent:b2w/0.1
Disallow:/

User-agent:copernic
Disallow:/

User-agent:psbot
Disallow:/

User-agent:python-urllib
Disallow:/


User-agent:netmechanic
Disallow:/

User-agent:url_spider_pro
Disallow:/

User-agent:cherrypicker
Disallow:/

Editor: Windear
Starting: SEO, SEM conspiracy, webmaster Resource Information Network
Address: http://www.seoplot.com,/
Reprint please source and keep the connection! Otherwise it will be

User-agent:emailcollector
Disallow:/

User-agent:emailsiphon
Disallow:/

User-agent:webbandit
Disallow:/

User-agent:emailwolf
Disallow:/

User-agent:extractorpro
Disallow:/

User-agent:copyrightcheck
Disallow:/

User-agent:crescent
Disallow:/

User-agent:sitesnagger
Disallow:/

User-agent:prowebwalker
Disallow:/

User-agent:cheesebot
Disallow:/

User-agent:lnspiderguy
Disallow:/

User-agent:mozilla
Disallow:/

User-agent:mozilla
Disallow:/

User-agent:mozilla/3
Disallow:/

User-agent:mozilla/4
Disallow:/

User-agent:mozilla/5
Disallow:/

User-agent:webauto
Disallow:/

User-agent:thenomad
Disallow:/

User-agent:www-collector-e
Disallow:/

User-agent:rma
Disallow:/

User-agent:libweb/clshttp
Disallow:/

User-agent:httplib
Disallow:/

User-agent:turingos
Disallow:/

User-agent:infonavirobot
Disallow:/

user-agent:harvest/1.5
Disallow:/

User-agent:crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/

user-agent:cherrypickerse/1.0
Disallow:/

user-agent:cherrypickerelite/1.0
Disallow:/

user-agent:webbandit/3.50
Disallow:/

User-agent:nicerspro
Disallow:/

User-agent:dittospyder
Disallow:/

User-agent:foobot
Disallow:/

User-agent:botalot
Disallow:/

user-agent:lwp-trivial/1.34
Disallow:/

User-agent:lwp-trivial
Disallow:/

User-agent:urly Warning
Disallow:/

User-agent:hloader
Disallow:/

User-agent:humanlinks
Disallow:/

User-agent:linkextractorpro
Disallow:/

User-agent:offline Explorer
Disallow:/

User-agent:mata Hari
Disallow:/

User-agent:lexibot
Disallow:/

User-agent:web Image Collector
Disallow:/

User-agent:the intraformant
Disallow:/

user-agent:true_robot/1.0
Disallow:/

User-agent:true_robot
Disallow:/

user-agent:blowfish/1.0
Disallow:/

User-agent:jennybot
Disallow:/

user-agent:miixpc/4.2
Disallow:/

User-agent:builtbottough
Disallow:/

user-agent:propowerbot/2.14
Disallow:/

user-agent:backdoorbot/1.0
Disallow:/

User-agent:tocrawl/urldispatcher
Disallow:/

User-agent:webenhancer
Disallow:/

User-agent:suzuran
Disallow:/

User-agent:vci webviewer VCI webviewer Win32
Disallow:/

User-agent:vci
Disallow:/

user-agent:szukacz/1.4
Disallow:/

User-agent:queryn metasearch
Disallow:/

User-agent:openfind Data Gathere
Disallow:/

User-agent:openfind
Disallow:/

User-agent:xenu ' s Link sleuth 1.1c
Disallow:/

User-agent:xenu ' s
Disallow:/

User-agent:zeus
Disallow:/

User-agent:repomonkey Bait & tackle/v1.01
Disallow:/

User-agent:repomonkey
Disallow:/

User-agent:openbot
Disallow:/

User-agent:url Control
Disallow:/

User-agent:zeus Link Scout
Disallow:/

User-agent:zeus 32297 Webster Pro V2.9 Win32
Disallow:/

User-agent:erocrawler
Disallow:/

User-agent:linkscan/8.1a Unix
Disallow:/

User-agent:keyword density/0.9
Disallow:/

User-agent:kenjin Spider
Disallow:/

user-agent:iron33/1.0.2
Disallow:/

User-agent:bookmark Search Tool
Disallow:/

user-agent:getright/4.2
Disallow:/

User-agent:fairad Client
Disallow:/

User-agent:gaisbot
Disallow:/

User-agent:aqua_products
Disallow:/

User-agent:radiation Retriever 1.1
Disallow:/

User-agent:webmasterworld Extractor
Disallow:/

User-agent:flaming Attackbot
Disallow:/

User-agent:oracle Ultra Search
Disallow:/

User-agent:msiecrawler
Disallow:/

User-agent:perman
Disallow:/

User-agent:searchpreview
Disallow:/

User-agent:naver
Disallow:/

User-agent:dumbot
Disallow:/

User-agent:hatena antenna
Disallow:/

User-agent:grub-client
Disallow:/

User-agent:grub
Disallow:/

User-agent:larbin
Disallow:/

user-agent:b2w/0.1
Disallow:/

User-agent:copernic
Disallow:/

User-agent:psbot
Disallow:/

User-agent:python-urllib
Disallow:/

User-agent:emailwolf
Disallow:/

User-agent:extractorpro
Disallow:/

User-agent:copyrightcheck
Disallow:/

User-agent:crescent
Disallow:/

User-agent:sitesnagger
Disallow:/

User-agent:prowebwalker
Disallow:/

User-agent:cheesebot
Disallow:/

User-agent:lnspiderguy
Disallow:/

User-agent:mister PiX
Disallow:/

User-agent:webauto
Disallow:/

User-agent:thenomad
Disallow:/

User-agent:www-collector-e
Disallow:/

User-agent:rma
Disallow:/

User-agent:httplib
Disallow:/

User-agent:turingos
Disallow:/

User-agent:infonavirobot
Disallow:/

user-agent:harvest/1.5
Disallow:/

user-agent:bullseye/1.0
Disallow:/

user-agent:mozilla/4.0 (compatible; Bullseye; Windows 95)
Disallow:/

User-agent:crescent Internet ToolPak HTTP OLE Control v.1.0
Disallow:/

user-agent:cherrypickerse/1.0
Disallow:/

user-agent:cherrypickerelite/1.0
Disallow:/

User-agent:nicerspro
Disallow:/

User-agent:urly Warning
Disallow:/

User-agent:humanlinks
Disallow:/

User-agent:web Image Collector
Disallow:/

User-agent:the intraformant
Disallow:/

user-agent:true_robot/1.0
Disallow:/

User-agent:true_robot
Disallow:/

user-agent:blowfish/1.0
Disallow:/

User-agent:jennybot
Disallow:/

user-agent:miixpc/4.2
Disallow:/

User-agent:builtbottough
Disallow:/

user-agent:propowerbot/2.14
Disallow:/

user-agent:backdoorbot/1.0
Disallow:/

User-agent:tocrawl/urldispatcher
Disallow:/

User-agent:webenhancer
Disallow:/

User-agent:suzuran
Disallow:/

User-agent:vci webviewer VCI webviewer Win32
Disallow:/

User-agent:vci
Disallow:/

user-agent:szukacz/1.4
Disallow:/

User-agent:queryn metasearch
Disallow:/

User-agent:openfind Data Gathere
Disallow:/

User-agent:openfind
Disallow:/

User-agent:xenu ' s Link sleuth 1.1c
Disallow:/

User-agent:xenu ' s
Disallow:/

User-agent:zeus
Disallow:/

User-agent:repomonkey Bait & tackle/v1.01
Disallow:/

User-agent:repomonkey
Disallow:/

User-agent:microsoft URL Control
Disallow:/

User-agent:openbot
Disallow:/

User-agent:url Control
Disallow:/

User-agent:zeus Link Scout
Disallow:/

User-agent:zeus 32297 Webster Pro V2.9 Win32
Disallow:/

User-agent:webster Pro
Disallow:/

User-agent:erocrawler
Disallow:/

User-agent:linkscan/8.1a Unix
Disallow:/

User-agent:keyword density/0.9
Disallow:/

User-agent:kenjin Spider
Disallow:/

user-agent:iron33/1.0.2
Disallow:/

User-agent:bookmark Search Tool
Disallow:/

user-agent:getright/4.2
Disallow:/

User-agent:fairad Client
Disallow:/

User-agent:gaisbot
Disallow:/

User-agent:aqua_products
Disallow:/

User-agent:radiation Retriever 1.1
Disallow:/

User-agent:webmasterworld Extractor
Disallow:/

User-agent:flaming Attackbot
Disallow:/

User-agent:oracle Ultra Search
Disallow:/

User-agent:msiecrawler
Disallow:/

User-agent:perman
Disallow:/

User-agent:searchpreview
Disallow:/

User-agent:sootle
Disallow:/

User-agent:es
Disallow:/

user-agent:enterprise_search/1.0
Disallow:/

User-agent:enterprise_search
Disallow:/

Editor: Windear
Starting: SEO, SEM conspiracy, webmaster Resource Information Network
Address: http://www.seoplot.com,/
Reprint please source and keep the connection! Otherwise it will be

This article is an English version of an article which is originally in the Chinese language on aliyun.com and is provided for information purposes only. This website makes no representation or warranty of any kind, either expressed or implied, as to the accuracy, completeness ownership or reliability of the article or any translations thereof. If you have any concerns or complaints relating to the article, please send an email, providing a detailed description of the concern or complaint, to info-contact@alibabacloud.com. A staff member will contact you within 5 working days. Once verified, infringing content will be removed immediately.

Related Keywords:
database search engines list list of javascript engines list of gem names names of the websites the names of the oceans how to be on the top of google search list list of linux os names

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

Hot Article
Hot Tags
computing conference access forum computer class data get http html applications
Popular Keywords
html add blank space register business logo register ssl certificate full site sign in sign up node js build cloud register register a subdomain in python network management system tutorial how to learn computer science by myself

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

Get Started for Free

  • Sales Support

    1 on 1 presale consultation

    Chat Contact Sales

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    Open a Ticket

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.

    Learn More