Search engine history-search engine technology

In the early days of Internet development, the site is relatively small, information lookup is easier. However, with the explosive development of the Internet, ordinary network users want to find the necessary information is like a needle in a haystack, then to meet the needs of the public information retrieval of professional search site has emerged.
The ancestor of the search engine in the modern sense was the Archie invented by Alan Emtage, a university student in Montreal in 1990. Although the World Wide Web has not yet appeared, but the network file transfer is quite frequent, and because a large number of files scattered in each of the scattered FTP host, the query is very inconvenient, so Alan Emtage thought of the development of a file name can be found in the system, And then there was Archie.
Archie work with the current search engine is very close, it relies on the script to automatically search the files on the web, and then index the information for users to a certain expression query. Inspired by the popularity of Archie, the Nevada system Computing Services University in the United States developed another very similar search tool in 1993, but the search tool has been able to retrieve web pages in addition to index files.
At the time, the word "robot" was very popular among programmers. The computer "robot" (Computer Robot) refers to a software program that can perform a task without interruption at a speed that humans cannot reach. Because the "robot" program for retrieving information is crawling around the web like a spider, the search engine's "robot" program is called the "Spider" Program.
The world's first "robot" program to monitor the size of the Internet is Matthew Gray's wide Web wanderer. In the beginning it was only used to count the number of servers on the Internet, and later it developed to be able to retrieve the domain name.
As opposed to Wanderer, Martin Koster created the Aliweb in October 1993, which is the HTTP version of Archie. Instead of using a "robot" program, Aliweb is relying on the site to proactively submit information to build its own linked index, similar to what we now know about Yahoo.
With the rapid development of the Internet, it is becoming more and more difficult to retrieve all the new appearing pages, therefore, on the basis of Matthew Gray's wanderer, some programmers have improved the traditional "spider" Program working principle. The idea is that since all pages may have links to other sites, it is possible to retrieve the entire Internet from the start of tracking links to one site. By the end of 1993, a number of search engines based on this principle began to emerge, with jumpstation, the World Wide Web Worm (the predecessor of Goto, which is today Overture), and repository-based Software Engineering (RBSE) spider the most prestigious.
However, jumpstation and www worm simply ranked search results in the search tool in order to find matching information in the database, so there was no information related to the degree. Rbse is the first engine to introduce the concept of keyword string matching in the search results arrangement.