Search engine Learning Resource Collection

I. Search Engine Technology/Dynamic Resources
<1> comprehensive
1. Research http://www.wespoke.com/
Lu Liang is a search engine development expert, previously developed a search engine "booso" (http://booso.com/), it seems that the development has been stopped, he is currently serving the blog. He can learn a lot about the technology and experience of search engine development on his blog, which deserves continuous attention.
2. laolu'blog
There are a lot of foreign information about search engines, focusing on data and numbers.
3. haslog http://www.loverty.org/
Here we can see the latest developments in several major search engines at home and abroad. It is worth looking at the search development situation.
4. Beijing yitian ruixin Technology Co., http://www.21cnbj.com/
News in search engines, Seo, SEM, and other industries
5. Chinese search engine guide http://www.sowang.com/
Latest Developments in search engines, various search techniques and methods
6. Full text search http://www.fullsearcher.com/
Fullsearcher. com was founded by two young people who are interested in searching. Our goal is to bring the Chinese internet into the search age and make search everywhere. Search to change people's lives.
Fullsearcher provides full-text search-related knowledge, vertical search engine knowledge, search-related news, and other search-related content.
7. Zhou Bo: Search Engine updates at every day
8. Li Yanhong's blog http://hi.baidu.com/liyanhong
9. The Chinese Emy of Sciences Software Institute-Zhang junlin blog Co., http://blog.csdn.net/malefactor/.
Research on Search Engine Technology
<2> Google News
Google official blog: Google Blackboard newspaper http://googlechinablog.com/
Google's blog in China is approaching our products, technologies, and culture.
1. gfans http://gfans.org/
2, G fast customer http://www.gseeker.com/
<2> other search engine Dynamics
1. Yahoo Search log http://ysearchblog.cn/
Record the dynamics, products, and technologies of Yahoo Search Engine
2. sogou lab http://www.sogou.com/labs/
Sogou Labs is a window for the core R & D team of sogou search to communicate with each other. It contains four topics: creative products, prototype demonstrations, data downloads, and academic papers. The lab warmly welcomes Internet players who have been paying attention to sogou and who have supported sogou. They are also eager to look forward to frequent visits from academic colleagues who are devoted to Chinese Internet research. We hope to use this platform to show the powerful R & D and innovation capabilities of sogou R & D team, promote interaction between academia and industry, and understand users' requirements for new products. Our goal is to provide more comprehensive and better services for Chinese Internet users.
Sogou lab blog http://labs.blog.sohu.com/
3. Baidu space http://hi.baidu.com/baidu
Baidu news
4, youdao search blog http:// I .yodao.com/
Netease's new search engine-Recent Developments in youdao search
5. Live Search's weblog http://blogs.msdn.com/livesearch/
Microsoft Live Search's news http://windowslivebeta.spaces.live.com/

Search engine Learning Resource Collection
I. Search Engine Technology/Dynamic Resources
<1> comprehensive
1. Research http://www.wespoke.com/
Lu Liang is a search engine development expert, previously developed a search engine "booso" (http://booso.com/), it seems that the development has been stopped, he is currently serving the blog. He can learn a lot about the technology and experience of search engine development on his blog, which deserves continuous attention.
2. laolu'blog
There are a lot of foreign information about search engines, focusing on data and numbers.
3. haslog http://www.loverty.org/
Here we can see the latest developments in several major search engines at home and abroad. It is worth looking at the search development situation.
4. Beijing yitian ruixin Technology Co., http://www.21cnbj.com/
News in search engines, Seo, SEM, and other industries
5. Chinese search engine guide http://www.sowang.com/
Latest Developments in search engines, various search techniques and methods
6. Full text search http://www.fullsearcher.com/
Fullsearcher. com was founded by two young people who are interested in searching. Our goal is to bring the Chinese internet into the search age and make search everywhere. Search to change people's lives.
Fullsearcher provides full-text search-related knowledge, vertical search engine knowledge, search-related news, and other search-related content.
7. Zhou Bo: Search Engine updates at every day
8. Li Yanhong's blog http://hi.baidu.com/liyanhong
9. The Chinese Emy of Sciences Software Institute-Zhang junlin blog Co., http://blog.csdn.net/malefactor/.
Research on Search Engine Technology
<2> Google News
Google official blog: Google Blackboard newspaper http://googlechinablog.com/
Google's blog in China is approaching our products, technologies, and culture.
1. gfans http://gfans.org/
2, G fast customer http://www.gseeker.com/
<2> other search engine Dynamics
1. Yahoo Search log http://ysearchblog.cn/
Record the dynamics, products, and technologies of Yahoo Search Engine
2. sogou lab http://www.sogou.com/labs/
Sogou Labs is a window for the core R & D team of sogou search to communicate with each other. It contains four topics: creative products, prototype demonstrations, data downloads, and academic papers. The lab warmly welcomes Internet players who have been paying attention to sogou and who have supported sogou. They are also eager to look forward to frequent visits from academic colleagues who are devoted to Chinese Internet research. We hope to use this platform to show the powerful R & D and innovation capabilities of sogou R & D team, promote interaction between academia and industry, and understand users' requirements for new products. Our goal is to provide more comprehensive and better services for Chinese Internet users.
Sogou lab blog http://labs.blog.sohu.com/
3. Baidu space http://hi.baidu.com/baidu
Baidu news
4, youdao search blog http:// I .yodao.com/
Netease's new search engine-Recent Developments in youdao search
5. Live Search's weblog http://blogs.msdn.com/livesearch/
Microsoft Live Search's news http://windowslivebeta.spaces.live.com/
2. Search Engine Code Resources
1> Search Engine/web spider program code
Related Programs developed abroad
1. nutch
Official Website http://www.nutch.org/
Chinese site http://www.nutchchina.com/
Latest Version: nutch 0.7.2 released
Nutch is a search engine implemented by open-source Java. It provides all the tools we need to run our own search engine. you can create your own search engine on the Intranet, or you can create a search engine on the entire network. Free and free ).
2. Lucene
Official Website http://lucene.apache.org
Chinese site http://www.lucene.com.cn/
Lucene is a sub-project of the Jakarta Project Team of the Apache Software Foundation. It is an open-source full-text search engine toolkit [written in Java], that is, it is not a complete full-text search engine, it is a full-text search engine architecture that provides a complete query engine and index engine, some text analysis engines (English and German ). Lucene aims to provide software developers with a simple and easy-to-use toolkit to conveniently implement full-text retrieval in the target system, or build a complete full-text retrieval engine based on this.
3, larbin: http://larbin.sourceforge.net/index-eng.html
Larbin is an open-source Web Crawler/web spider, developed independently by French young man s é Bastien ailleret. Larbin aims to track the URLs of pages for extended crawling, and finally provides a wide range of data sources for search engines.
Related Programs developed in China
1. sqlet-Open Source Chinese Search Engine
Official Website http://www.sqlet.com/
Sqlet, Which is search & query & link, with the suffix "let", indicating small and small meanings. we plan to build a topic-based Chinese search engine that can search hundreds of millions of webpages. three indexing methods are supported: mysql_table_index, e_e_index, and sqlet_index. web page capture can be stored in the file system and database. Webserver.
2. findu vertical search engine code
Feidu http://www.faydu.net is a demonstration version of vertical online search, mainly for some domestic shopping sites to search,
The code in the open-source beta version is available for discussion. Download instructions:
1. Because the program runs on the server and runs on multiple processors, please control the number of threads on your PC.
2. Restore a database containing data to SQL Server
3. After the collection is completed, the reverse index file generated by licene is displayed in the bin directory by default.
4 ": http://www.faydu.net/download/code.rar
Open Date: Source: http://blog.csdn.net/faydu/archive/2006/04/18/667997.aspx
Language: VB.net (C #)
2> Chinese word segmentation program code
1. Calculate the Chinese lexical analysis system ICTCLAS
Based on years of research, the Institute of Computing Technology of the Chinese Emy of Sciences developed the Chinese lexical analysis system ICTCLAS (Institute of computing technology, Chinese Lexical Analysis System) based on the multilayer hidden horse model ), the system has the following functions: Chinese word segmentation, part-of-speech tagging, and unregistered word recognition. The word segmentation accuracy rate is as high as 97.58% (the most recent 973 Expert Group evaluation results). Role-Based unregistered word recognition can achieve a recall rate higher than 90%, of which the recall rate of Chinese names is close to 98%, the processing speed of Word Segmentation and part-of-speech tagging is 31.5kb/s. The results of 14 free ICTCLAS and 14 other computing institutes were widely reported by Chinese and foreign media. Many free Chinese Word Segmentation modules in China have referenced ICTCLAS code more or less.
Download: http://www.nlp.org.cn/project/project.php? Proj_id = 6
Since ICTCLAS is written in C language, it is not convenient to use mainstream development tools. Therefore, some enthusiastic programmers have changed ICTCLAS to other languages such as Java and C.
(1) fenci, Java ICTCLAS, download page: http://www.xml.org.cn/printpage.asp? Boolean id = 2 & id = 11502
(2) autosplit, another Java ICTCLAS, cannot find the download page. Click Download locally
(3) Xiao Ding-dong's Chinese word segmentation has been found on the download page and cannot be found now. According to the author, improvements from ICTCLAS, there are Java, C # And C ++ three versions, Introduction page: http://www.donews.net/accesine
2. Massive smart word segmentation research Edition
The massive intelligent Computing Technology Research Center aims to enable researchers in the Chinese Information Processing field to share the research results of the massive intelligent center and jointly improve the level of Chinese Information Processing, we hereby release the massive Intelligent Word Segmentation research edition for research by experts, scholars and enthusiasts.
Download: http://www.hylanda.com/cgi-bin/download/download.asp? Id = 8
3. Others
(1) CSW Chinese Intelligent Word Segmentation component
Running Environment: Windows NT, 2000, XP, or higher, which can be called in Microsoft development languages such as ASP and VB.
Introduction: CSW intelligent Chinese Word Segmentation DLL component, which can automatically split a piece of text into regular Chinese phrases and separate them in a specified way, the split phrases can be marked with semantics and word frequency. It is widely used for information retrieval and analysis in various industries.
Download Page: http://www.vgoogle.net/
(2) C # Chinese Word Segmentation component written
According to the author, a DLL file can be used as a Chinese/English word splitting component. Fully C # managed code writing and independent development.
Download: http://www.rainsts.net/article.asp? Id = 48
Edit reference report rating reply top
View IP address dongdonglang
Moderator
Rank: 7 rank: 7 rank: 7
UID 1000
Essence 0
Points 39
Post 11
Read Permission 100
Register
Online status
#3
Short Message published on PM
3> overview of open-source spider
Spider is a required module for search engines. The results of spider data directly affect the evaluation indicators of search engines.
The first Spider Program was operated by MIT's Matthew K gray to count the number of hosts on the Internet.
Spier definition (there are two definitions of spider: broad and narrow ).
* Narrow sense: software programs that traverse the World Wide Web information space using the standard HTTP protocol based on the hyperlink and web document retrieval methods.
* Broadly speaking, all software that can use HTTP to Retrieve Web documents is called Spider.
Protocol gives sites way to keep out of the 'bots Jeremy Carl, Web week, Volume 1, Issue 7, November 1995 is a protocol closely related to Spider. For details, refer to robotstxt.org.
Heritrix
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
Heritrix (sometimes spelled heretrix, or misspelled or missaid as heratrix/heritix/heretix/heratix) is an archaic word for heiress (woman who inherits ). since our crawler seeks to collect and preserve the Digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 73833 & package_id = 73980
Weblech URL spider
Weblech is a fully featured web site download/mirror tool in Java, which supports many features required to download websites and emulate standard web-browser behaviour as much as possible. weblech is multithreaded and comes with a GUI console.
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 38170
Jspider
A Java implementation of a flexible and extensible web spider engine. optional modules allow functionality to be added (searching dead links, testing the performance and scalability of a site, creating a sitemap, Etc ..
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 65617
Websphinx
Websphinx is a web crawler (robot, spider) Java class library, originally developed by Robert Miller of Carnegie Mellon University. multithreaded, tollerant HTML parsing, URL filtering and page Classification, pattern matching, grouping, and more.
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 48810
Pysolitaire
Pysolitaire is a fork of pysol solitaire that runs correctly on Windows and has a nice clean installer. Equals (Python solitaire) is a collection of more than 300 solitaire and mahjongg games like Klondike and spider.
Language ython, () http://sourceforge.net/project/showfiles.php? Group_id = 86107
The spider web network xoops mod team
The spider web network xoops module team provides modules for the xoops community written in the PHP coding language. we develop mod and or take existing PHP script and port it into the xoops format. high quality mod is our goal.
Language HP, () http://sourceforge.net/projects/tswnmoddev
Fetchgals
A multi-threaded web spider that finds free porn thumbnail galleries by visiting a list of known tgps (thumbnail gallery posts ). it optionally downloads the located pictures and movies. TGP list is supported ded. public Domain Perl script running on Linux.
Language ERL, () http://sourceforge.net/project/showfiles.php? Group_id = 110338
Where spider
The purpose of the where Spider software is to provide a database system for storing URL addresses. the software is used for both ripping links and browsing them offline. the software uses a pure XML database which is easy to export and import.
Language: XML, () http://sourceforge.net/project/showfiles.php? Group_id = 115931
Sperowider
Sperowider website archiving suite is a set of Java applications, the primary purpose of which is to Spider dynamic websites, and to create static distributable archives with a full text search index usable by an associated Java applet.
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 90254
Spiderpy
Spiderpy is a web crawler spider program written in Python that allows users to collect files and search Web sites through a writable able interface.
Language ython, () http://sourceforge.net/project/showfiles.php? Group_id = 55531
Spidered Data Retrieval
Spider is a complete standalone Java application designed to easily integrate varied CES. * XML driven framework * Scheduled pulling * highly extensible * provides hooks for custom post-processing and Configuration
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 90769
Webloupe
Webloupe is a Java-based tool for analysis, interactive visualization (sitemap), and isolation of the Information Architecture and specific properties of local or publicly accessible websites. based on Web Spider (or web crawler) technology.
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 121963
Aspider
Robust featureful multi-threaded CLI web spider using Apache commons httpclient V3.0 written in Java. aspider downloads any files matching your given mime-types from a website. tries to Reg. exp. match emails by default, logging all results using log4j.
Language: Java, () http://sourceforge.net/project/showfiles.php? Group_id = 126578
Larbin
Larbin is an HTTP web crawler with an easy interface that runs under Linux. It can fetch more than 5 million pages a day on a standard PC (with a good network ).
Language: C ++, () http://sourceforge.net/project/showfiles.php? Group_id = 42562
Iii. Seo-related resources
1. Domain Name Information Query
★Query the information of the international top-level domain name (. aero ,. arpa ,. biz ,. com ,. coop ,. edu ,. info ,. int ,. museum ,. net ,. can be queried through the domain name registrar authorized by ICANN, or directly to the InterNIC website, the URL is
Http://www.internic.com/whois.html
Http://www.iwhois.com/
★To query whether a global top-level domain name has been registered, you can go to the following website (including the domestic domain name. cn ):
Http://www.uwhois.com/cgi/domains.cgi? User = noads
★Query the registration status of domestic domain names,
Http://ewhois.cnnic.net.cn/index.jsp
★Hichina domain name registration information query
Http://www.net.cn/
★IP address query and domain name registration information whois Query
Http://ip.zahuopu.com/
2. Alexa-related and search rankings
★Ranking Top 500 in Chinese
Http://www.alexa.com/site/ds/top... & lang = zh_gb2312
★Google Zeitgeist-Google search rankings
Http://www.google.com/press/intl-zeitgeist.html#cn
★Baidu Chinese search rankings
Http://top.baidu.com/
★Yahoo search rankings
Http://misc.yahoo.com.cn/top_index.html
★Sogou search index
Http://www.sogou.com/top/
3. Search for keywords
★Google Keyword query https://adwords.google.com/select/KeywordSandbox
★Baidu keyword query http://www2.baidu.com/inquire/dsquery.php
★Sohu key words http://db.sohu.com/regurl/pv_price/query_consumer.asp
4. External link query
★Sogou link query http://www.sogou.com/features/
5. Seo projects/tools
★Web Quality http://category.booso.com/cgi-bin/category/category.cgi
★Keywords density http://www.21ql.com/seo/keyword.asp
★Search engine spider simulator Co., http://www.webconfs.com/search-engine-spider-simulator.php.
★Google dance query tool: http://www.google-dance-tool.com/
6. Seo websites
English website:
Search observation http://www.searchenginewatch.com/
Seochat http://www.seochat.com
Chinese website
The latest SEO industry information can be obtained here
Search engine optimization exchange center http://www.seoonline.cn