Chinese search engine technology unveiling: System Architecture (4)

Source: e800.com.cn

Search engine index and search For the network spider technology and Sorting Technology, refer to the author's otherArticle[1] [2]. Here we take the Google search engine as an example to introduce the data indexing and search process of the search engine. Data Indexing consists of three steps: webpage content extraction, word recognition, and indexing database establishment. Most information on the Internet exists in HTML format. For indexes, only text information is processed. Therefore, we need to extract the text content from the webpage, filter out some script identifiers and useless advertisement information, and record the layout information of the text [1]. Word Recognition is a key part of search engines. It uses dictionary files to recognize words in webpages. For Spanish information, it is necessary to identify different forms of words, such as single and multiple numbers, past tense, composite words, and word roots. For some Asian languages (Chinese, Japanese, and Korean) word Segmentation is required [3]. Identifies each word in the webpage and assigns a unique wordid to serve the indexing module in the data index. The indexing database is the most complex part of the data index structure. Two indexing methods are generally required: document indexing and keyword indexing. Document Indexing allocates a unique docid number for each web page. The number of times, location, and Case format of each wordid appear in this web page according to the docid, form a data list of the wordid corresponding to the docid. keyword indexing is actually a reverse indexing of the document. This word is derived from the wordid mark on those webpages (represented by wordid ), the number of times, locations, and Case formats displayed on each webpage form a list of docids corresponding to the wordid. For more information about the detailed data structure of index data, see [4]. The search processing process is the process of satisfying the user's search request. By entering the search keyword, the search server searches for the corresponding keyword dictionary and converts the search keyword to the wordid, then, obtain the docid list in the index database, scan the docid list and match the wordid, extract the qualified webpage, and then calculate the relevance between the webpage and keywords, return the first K results based on the correlation value (different search engines return different results per page) to the user. If you search for the second or second page, the page organization in the sorting result from k + 1 to 2 * k is returned to the user. The process is shown in: