Search engine technical articles

Search Engine TechnologyArticle
Xie Xinbo http://blog.xieyubo.com/

SF: open-source FTP Search Engine
Http://gf.cs.hit.edu.cn
Related Documents
Supersonic version
Pay attention to some details and letProgramFaster running (1/4)
Pay attention to some details to make the program run faster (2/4)
Pay attention to some details to make the program run faster (3/4)
Pay attention to some details to make the program run faster (4/4)
SF supersonic version data structure (1/3)
SF supersonic version data structure (2/3)
SF supersonic version data structure (3/3)
Supersonic version
SF sub-sonic Version System Architecture (1/3)
SF sub-sonic Version System Architecture (2/3)
SF sub-sonic Version System Architecture (3/3)
SF search engine-IP source statistics development document

BaiduAlgorithm-Query Processing and Word Segmentation technologyHttp://hi.baidu.com/jiewangzi/blog/item/0e7bc23593e81d1390ef3936.html
Query Processing/Chinese word segmentation.
Now the word segmentation algorithm is relatively mature and simple and complex, such as forward maximum matching, reverse maximum matching, bidirectional maximum matching, language model method, and shortest path algorithm, if you are interested, you can search by Google to increase understanding.

 A simple Chinese word segmentation model using the forward maximum matching algorithm-implemented using the trie tree
Http://blog.csdn.net/lyflower/archive/2006/12/21/1452091.aspx

Search engine Cache Policy Research
Http://software.hit.edu.cn/eestudio/bbs/ShowPost.asp? Threadid = 271

 

Mapreduce: Simplified data processing on large clusters
Jeffrey Dean and Sanjay Ghemawat
Http://labs.google.com/papers/mapreduce.html

Abstract

Mapreduce is a programming model and an associated implementation for processing and generating large data sets. users specify a map function that processes a key/value pair to generate a set of intermediate key/value pairs, and a reduce function that merges all intermediate values associated with the same intermediate key. using Real World tasks are expressible in this model, as shown in the paper.

Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines. the run-time system takes care of the details of partitioning the input data, scheduling the program's execution processing ss a set of machines, handling machine failures, and managing the required inter-machine communication. this allows programmers without any experience with parallel and distributed systems to easily utilize the resources of a large distributed system.

Our implementation of mapreduce runs on a large cluster of commodity machines and is highly scalable: a typical mapreduce computation processes extends terabytes of data on thousands of machines. programmers find the system easy to use: hundreds of mapreduce programs have been implemented and upwards of one thousand mapreduce jobs are executed on Google's clusters every day.

Appeared in:
Osdi '04: Sixth Symposium on operating system design and implementation,
San Francisco, CA, December, 2004.

Download: PDF version

Slides: HTML slides