Search engine implementation based on Linux in PHP

Source: Internet
Author: User
The search engine provides users with a tool to quickly obtain Web page information. its important function is that the system retrieves the back-end web database by entering key words, feed back the links and abstract information of the relevant webpage to the user. The search scope is generally divided

The search engine provides users with a tool to quickly obtain Web page information. its important function is that the system retrieves the back-end web database by entering key words, feed back the links and abstract information of the relevant webpage to the user. In terms of search scope, it is generally divided into intra-site web search and global web search. With the sharp increase in the number of web pages, search engines have become an essential means of querying information on the Internet, and various large websites have provided webpage data search services, in addition, many companies provide professional search engine services for large websites, such as Google, Baidu, which provides services for domestic websites such as Sina and 263. Professional search services are highly expensive and free search engine software is based on English retrieval, so it is not suitable for the needs of Intranet environments (such as campus networks.

The basic components of a search engine are generally divided into three parts: Web page collection program, web page back-end data organization and storage, and web page data retrieval. The key factor that determines whether a search engine is good or bad is the response time of the data query, that is, how to organize a large number of web page data to meet the full-text search needs.

As an excellent network control system, GNU/Linux has integrated a large number of network utilization software, such as Web servers (Apache + PHP) and directory servers (OpenLDAP) script language (Perl), web page collection program (Wget), etc. Therefore, you can implement a simple and efficient search engine server by using them in a centralized manner.

  I. basic components and application methods

1. webpage data collection

Wget is an excellent web page collection program. it uses multi-threaded design to easily mirror website content to a local directory, in addition, you can customize the collection page type, progressive collection level, directory quota, and collection time. Using a dedicated collection program to collect webpages not only reduces the design difficulty but also improves the system performance. To reduce the scope of local data, you can only collect html files, txt files, script programs asp and php that can be queried and only apply the default results, instead of collecting shape files or other data files.

2. webpage data filtering

Because a large number of tags exist in html files, such

3. directory service

The Directory service is a service that needs to be developed for a large number of data searches. it was first presented in the X.500 Protocol set and later expanded to TCP/IP to become the Lightweight Directory Acess Protocol, the coherent scale is the RFC1777 formulated in 1995 and the rfc2133 formulated in 1997. LDAP protocol has been widely used in related products by Sun, Lotus, Microsoft, and other companies as industrial standards. however, dedicated directory servers based on Windows platforms are rare, openLDAP is a directory server running on Unix systems free of charge. It provides excellent performance and has been collected by many Linux distributions (such as Redhat and Mandrake ), it also provides development interfaces including C, Perl, and PHP.

These labeled data has no actual search value. Therefore, you must filter the collected data before participating in the database. As a widely used scripting language, Perl has a very strong and rich library that can easily filter webpages. The HTML-Parser library can be used to conveniently extract text data, title data, and link data contained in a webpage. This library can be downloaded from www.cpan.net, and the Perl programs collected by this website involve a wide range, far beyond our phenomenon.

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.