How does the SharePoint search function work in a multi-language environment?

How can I enable normal indexing and query of content in a language in multiple languages?

What are the principles for indexing and querying site content, Office documents, and TXT documents in different language environments?

 

The text below is a piece of White Paper from SharePoint 2003.

After the author's test, this text description of the multilingual search function is still correct for SharePoint 2010.

 

How Search Service Manages Multilingual Contents

========================================================== ===

The following sections provide information on how the specified pointpssearch service treats content at index time and at query time, depending on the type of the content, its location, and user language settings.

Language Considerations for SharePoint Portal Server Search Service

-----------------------

Indexing

• For clients that have a locale ID (LCID) and a language ID that are supported by the specified pointpssearch service (suchMicrosoft Office documents), The specified pointpssearch service uses the locale ID and a language ID that is associated with the document to identify the language resources that are used to index the document.
• For documents that have an LCID andLanguage ID that are not supportedBy the specified pointpssearch service, the neutral word breaker is used to index the document.
• For documents that use. TxtExtension, the specified pointpssearch service uses the default locale of the indexing server to index the content.
• For Windows SharePoint ServicesSitesThat are in the Site Directory, the specified pointpssearch service uses the language of the Windows SharePoint Services site.
• ForSitesBased on SharePoint Portal Server, the specified pointpssearch service uses the language of the site.

Querying

• Where possible, the SharePoint Portal Server Search service tries to identify the language of the user by using eitherDefault language of the user's Web browser, Or by using the optional LCID parameter in the query string to determine the language resources that are used to break the query terms.

If the language ID is not supported, the neutral language resources are used.

Language Considerations for Windows SharePoint Services Search Service

-------------------

Indexing

• For text that is stored directly in list elements (and not stored in documents), the column language is used to determine the language resources that are used for indexing.
• For documents that are stored in document libraries, the specified pointpssearch service indexing rules apply.

Querying

The default language of the column is used to determine the language resources that are used to break the query terms.

========================================================== ==========================================

The text below comes from another white paper named Plan for building multilingual solutions. Its Explanation of the search Component and process is classic.

 

Overview of the Language Features in Search

====================

Content crawling and querying are different processes provided by Search. Each process occurs at different times and uses resources in different ways.

Content Crawler

The index engine uses a pipe of shared memory to request thatFilter DaemonBegin filtering the content source. For the crawl process to succeed, the content source must have an associatedProtocol handlerThat can read its protocol.

The Filter Daemon invokes the appropriate protocol handler for the content source based on the start address provided by the index engine. the Filter Daemon uses protocol handlers and IFilters to extract and filter individual items from the content source. Appropriate IFilters for each document are applied, and the Filter Daemon passes the extracted text and metadata to the index engine through the pipe.

At this point, the index engine saves the document properties to a property store separate from that of the content index. the property store consists of a table of properties and their values. the properties in this store can be retrieved and sorted as needed. in addition, simple queries against the properties are supported by the store.Each row in the table corresponds to a separate document in the full-text index.The actual text of a content item is stored in the content index, so it can be used for content queries. the property store also maintains and enforces document-level security that are gathered when a document is crawler. after the initial crawl, the index engine uses word breakers and stemmers to further process the text and properties gathered during the crawl. the word breaker component is used to split the text into logical words and phrases. the index engine also removes "noise words" (that is, words that do not add value to a query) and creates an inverted index for full-text searching.

Search query execution

When a search query is executed for a given language, the query engine passes that query to a word breaker for that language.

If there is no word breaker for the language of the query, the neutral word breaker is used; the neutral word breaker uses white spaces to split the words.

After the word-breaking process, results are passed, optionally, through a language-specific stemmer (when one is available ). the stemming component is used to generate inflected forms of a given word, such as from a plural to a singular ("dogs" versus "dog ") or the tense of a verb ("spoke" versus "spoken ").

The use of the word breaker in both the crawler and query processes enhances the specified tiveness of Search because more relevant alternatives to a user's query phrasing can be generated.

When the query engine executes a property value query, the index is not touched and the query is saved med directly against Microsoft SQL Server where properties are stored to ensure a proper match.

The result of the query is a list of all matching statements, ordered by their relevance to the query words. if the user does not have permissions to a matching document, the query engine removes that document from the returned list.

The following dimo-provides a detailed view of the language features in Search at both index and query time:

Language-specific Features Provided and Used by the Search Service

==============

Word breakers

A word breaker is a component used by the query and index engines to break compound words and phrases into individual words or tokens. if there is no word breaker for a specific language, the neutral word breaker is used, in which case word breaking occurs where there are white spaces between the words and phrases. at indexing time, if there is any locale information associated with the document (for example, a Word document contains locale information for each text chunk ), the index engine will try to use the word breaker for that locale. if the document does not contain any locale information, the user locale of the computer the indexer is installed on is used instead. at query time, the locale (HTTP_ACCEPT_LANGUAGE) of the browser from which the query was sent is used to perform word breaking on the query. additional information about the language availability of the word breaker component is available inAppendix B: Search Language Considerations.

Noise words dictionary

Noise words are words that do not add value to a query, such as "and," the, "and". "The indexing engine filters them to save index space and to increase performance. noise word files are customizable, language-specific text files. these files are a simple list of words, one per line. if a noise word file is changed, you must perform a full update of the index to induplicate ate the changes. additional information about the noise words dictionary and how to mimize it is available at www.microsoft.com.

Custom dictionary

The custom dictionary file contains values that the search server must include at index and query times. custom dictionary lists are customizable, language-specific text files. these files are used by Search in both the index and query processes to identify exceptions to the noise word dictionaries. A word such as "AT&T," for example, will never be indexed by default because the word breaks it into single noise words. to avoid this, the user can add "AT&T" to the custom dictionary file; as result, this word will be treated as an exception by the word breaker and will be indexed and queried. these files contain a simple list of words, one per line. if the custom dictionary file is changed, you must perform a full update of the index to induplicate ate the changes. by default, no custom dictionary file is installed during Office SharePoint Server 2007 Setup. additional information about the custom dictionary file and how to mimize it is available at www.microsoft.com.

Thesaurus

There is a retriable thesaurus file for each language that Search supports. using the thesaurus, you can specify synonyms for words and also automatically replace words in a query with other words that you specify. the thesaurus used will always be in the language of the query, not necessarily the server's user locale. if a language-specific thesaurus is not available, a neutral thesaurus (tseneu. xml) is used. additional information about the thesaurus file and how to customize it is available at www.microsoft.com.

References:

Whitepaper (legacy): Using SPS 2003 in multilingual scenarios

White paper: Plan for building multilingual solutions

Http://technet.microsoft.com/en-us/library/cc262942%28office.12%29.aspx