The implementation of the search engine based on Linux
Search engine is to provide users with quick access to web information tools, the main function is the system through the user input keywords, search back-end Web database, the relevant web page links and summary information feedback to the user. From the scope of the search is generally divided into site search and global web search. With the rapid increase in the number of Web pages, search engines have become the means to access information online, each large web site has provided web data search services, and there have been many for large sites to provide professional search engine services, such as Google for Yahoo to provide search services, For Sina and 263 and other domestic websites to provide services, such as Baidu company. Professional search service costs high and free search engine software is basically based on English search, so it is not suitable for intranet environment (such as campus network, etc.) needs.
The basic composition of search engine is generally divided into three parts: Web-page collection program, Web page back-end data organization storage, Web page data retrieval. The key factor determining the search engine quality is the response time of the data query, that is how to organize the large amount of Web page data to satisfy the Full-text search.
Gnu/linux as an excellent network operating system, its distribution has integrated a large number of network applications, such as Web server (Apache + PHP), directory Server (OpenLDAP), scripting language (Perl), Web page collection program (Wget). So, by concentrating them on the application, we can realize a simple and efficient search engine server.
I. Basic composition and methods of use
1. Web Data collection
Wget program is an excellent web collection program, it uses multithreading design can easily mirror the content of the site to the local directory, and can flexibly customize the collection of Web page types, recursive collection levels, directory limits, collection time and so on. Through the dedicated collection program to complete the collection of Web pages, both reduce the difficulty of the design and improve the performance of the system. To reduce the size of local data, you can only collect HTML files, txt files, scripts, ASP, and PHP using only the default results, rather than collecting graphics files or other data files.
2. Web Data filtering
Because of the large number of tags in HTML files, such as <body><table>, which have no actual search value, the data collected must be filtered before you join the database. Perl, as a widely used scripting language, has a very powerful and rich library of programs to easily filter Web pages. By using the Html-parser library, the text data, header data, link data and so on can be easily extracted from the Web page. The library can be downloaded in www.cpan.net, and the Perl program that the Web site collects covers a wide range of events, far beyond our own.
3, Directory Services
The directory service is a service that needs to be developed for a large number of data retrieval, first appearing in the X.500 protocol set, and later extended into TCP/IP to develop into an LDAP (lightweight Directory acess Protocol) protocol, Its relevant standards for the 1995 RFC1777 and 1997 developed the RFC2251 and so on. The LDAP protocol has been widely used by companies such as Sun, Lotus and Microsoft in their related products as industrial standards, but the dedicated Windows platform directory server is rare, OpenLDAP is a free directory server running on UNIX systems, and its products have excellent performance, has been collected by a number of Linux distributions (Redhat, Mandrake, etc.) and provides development interfaces including C, Perl, PHP, and so on.
