Analysis of SOLR distributed search technology

Overview

SOLRThere is a certain amount of data to be searched on a single machine, depending on the complexity of the search, server hardware configuration and business requirements, etc, therefore, the distribution of search functions will be an inevitable trend for big data search.

SOLRSlave1.3When the version starts, it comes with a distributed search (Distributed search). This function enablesSOLRSupports horizontal scaling through multiple servers and horizontal splitting of data to support searching for massive data volumes.

Solr-3.6.1The version supports the following distributed search functions:

Search function module	Support for distributed search
Query component	Y
Facet component	Y
Highlighting component	Y
Spell check component	Y
Terms component	Y
Stats component	Y
Term vector component	Y
Debug component	Y
Grouping component	Y
Queryelevationcomponent	N
Morelikethis	N
Join	N

Due to business functions and time, this article will only discussQuery componentTechnical Implementation logic.

Notes

In useSOLRPay attention to the following details during distributed search:

• Schema. xmlDefined inUnique keyMust be saved in the index. BecauseSOLRIn progress2nd phraseYou need to use thisUnique keyPerform Secondary confirmation of data consistency and obtain the field data that the search requires to query.
• The index files distributed on different servers containUnique keyDo not repeat. BecauseSOLRIn progress1st phraseYou needUnique keySort and remove duplicates, ifUnique keyThere are duplicates, including the sameUnique keyOfDocThe result is returned randomly.
• Do not flip too many pages in the search results. BecauseSOLRIn the distributed search, all the summary results that need to be sorted by page are returnedProxy SOLR ServerSort by summary. If there are too many pages, the network bandwidth will be under a certain amount of pressure.
• Note:HTTPNumber of connections. BecauseSOLRIn distributed search, the server may beSearch ServerAgainProxy Server, Wait againHTTPRequest Response is processed onceHTTPA deadlock may occur between multiple servers.

Distributed search logic implementation

Query componentThe implementation principle is:Multi-phased approach, allowing for inconsistencyThe specific implementation details are as follows:

2. The client sends a search requestSOLRAny server in the ClusterSP.
4. SPThe server processes distributed query requests.
2. Phase one
1. Build a query request to retrieve only the queryDocOfUnique keyAndSort FieldField.
2. Pass the constructed requestHTTPSend to eachSOLR shardNode.
3. WaitSOLR shardThe node returns the query result.
4. Merge one by one based on sorting rulesSOLR shardThe query result returned by the node.
4. Phase Two
1. construct a query request, query the field data queried by the client based on unique key .
2. use HTTP send the request to each SOLR shard node.
3. wait SOLR shard node return Query result.
4. merge one by one SOLR shard node return query results, build the final result of this query.
5. sp the server returns distributed query results to the client

Note: In the current version, if one ofShardException. The overall query will fail.

References

• Http://wiki.apache.org/solr/DistributedSearch
• Http://wiki.apache.org/solr/WritingDistributedSearchComponents
• Http://wiki.apache.org/solr/DistributedSearchDesign
• Solr-3.6.1Source code