What is Solrcloud? (And how does it compare to Master-slave?)

What is Solrcloud? (And how does it compare to Master-slave?)

Solrcloud is a set of new features and functionality added in SOLR 4.0 to enable a new from creating durable, highly AVA Ilable SOLR clusters with commodity hardware. While similar in many ways to Master-slave, Solrcloud automates a lot of the manual labor required in Master-slave through Using ZooKeeper nodes to monitor the state of the cluster as well as additional SOLR features that understand how to inte Ract with other machines in their SOLR cluster.

In short, where master-slave requires manual effort to change the role of a node in a given cluster and to add new nodes T o a cluster, Solrcloud aims to automate a lot of this work and allow seamless addition of the new nodes to a cluster and to Wo RK around downed nodes with minimal oversight.

Master-slave mode requires manual management of the node's role (Master/slave), Solrcloud can automatically perceive the state of the cluster (node hangs/resumes, or new node joins, etc.)

What does a solrcloud look like compared to Master-slave?

On the surface, the primary difference between Solrcloud and Master-slave are the additional requirement of at least three ZooKeeper (ZK) nodes. Effectively, this means the minimum size of a solrcloud cluster are larger than for Master-slave, and the ZK nodes do not n Eed to be particularly powerful. As their only role was to monitor and maintain the state of nodes in the Solrcloud, latency are more important than computin G Power so the ZooKeeper nodes can is fairly minimal machines, so long as they is dedicated to the purpose.

Why would I need Solrcloud or master-slave on all?

Solrcloud and Master-slave both address four particular issues:

• Sharding
• Near Real time (NRT) Search and incremental indexing
• Query Distribution and load balancing
• High Availability (HA)

What is sharding?

Sharding is the act of splitting a single SOLR index across multiple machines, also known as shards or slices in SOLR term Inology. Sharding is very often needed because an index have grown too large to fit in a single server. A given Shard can contain multiple physical/virtual servers, meaning that all the machines/replicas on that shard contain The same index data and serve queries for that data.

Through sharding, you can split your index across multiple machines and continue to grow without running into problems. More information can found Here:http://docs.lucidworks.com/display/solr/shards+and+indexing+data+in+solrcloud

Near Real time Search and incremental indexing:

Master-slave operates with every shard have a single master which takes care of all indexing. All other nodes in the Shard is slaves, and upon entering the Shard is given the current state of the index using Solrre Plication (http://wiki.apache.org/solr/SolrReplication). Once a slave have been replicated, the slave ' s pollinterval determines how often it would contact the Shard master to Receiv E index updates.

Solrcloud similarly have a leader in every shard but the leader are largely the same as any other replica, both indexing Documents and serving queries. The only additional responsibility of the leader are to distribute documents to being indexed to all other replicas in the SHA Rd, and to then the report of all replicas has confirmed receiving a given document. Any document sent into Solrcloud are re-routed to the leader of the appropriate Shard, who then performs this responsibilit Y. When a replica receives a document, it adds the document to its transaction log and it would send a response to the lead Er. In the This, the updates to Solrcloud indexes is performed in a distributed manner and is durable.

Once A document has been added to the transaction logs, it is available via a realtimeget (Http://wiki.apache.org/solr/Rea Ltimeget), but isn't available via search until a soft commit or hard commit with opensearcher=true have been executed. A Manual Soft Commit or hard commit would make all documents in the transaction log available for search. One can also use the autocommit and Autosoftcommit parameters to trigger commits from individual nodes on a regular basis.

Query Distribution and load balancing:

In Master-slave, querying a single node would only bring you results from the node, which in most cases is equivalent t o Querying one slice of your data. In order to generate a query for your entire sharded index, you must use a distributed Search (HTTP://WIKI.APACHE.ORG/SOLR /distributedsearch) to query one node per shard. In the event, the one of the those nodes are unable to respond, an error would be given and the query won't be fulfilled. This being the case, trying to load balance queries across a master-slave cluster can be problematic. Master-slave also does not provide no load balancing, and thus requires an external load balancer.

With Solrcloud, distributed searching are handled automatically by the nodes of the cloud:querying any node would cause That node-to-send the query out to one node in all other shards, returning a response if it has aggregated the resu Lts from all shards. Furthermore, ZooKeeper and all replicas is aware of any non-responding nodes, and therefore won ' t direct queries to nodes That is considered dead. In the event that a downed node have not yet been detected by ZK and is sent a query, the querying node would report the nod e as down-to-ZK, and resend the query to another node. In the This is the queries to a solrcloud is quite durable and would almost never be interrupted by a downed node.

Solrcloud can handle its own load balancing if you use a smart client such as SOLRJ (HTTP://WIKI.APACHE.ORG/SOLR/SOLRJ). SOLRJ would use a simple round robin load balancer, distributing queries evenly to all nodes in Solrcloud. Furthermore, SOLRJ is ZooKeeper aware and thus would never send a query to a node that's known as down.

High availability:

In the event of a downed master in a Master-slave cluster, the Shard can continue to serve queries, but would no longer be Able to index until a new master is instated. The process of promoting a slave to a master are manual, though it can be scripted. Any updates to the master's index since the last replication is lost, and those documents would have been resubmitted. When a slave disconnects from the cluster and then rejoins, it'll automatically retrieve any missed Updates/index Segmen TS from the master before it was considered ready to serve queries.

In Solrcloud, when ZooKeeper detects a leader have gone down, it'll initiate the leader election process instantaneous Ly, selecting a new leader to begin distributing documents again. Since the transaction log ensures that all nodes in the Shard is in sync, all updates is durable and never lost when a l Eader goes down. Similar to Master-slave, when a replica rejoins the cluster it simply replays the transaction log to bring itself up to Da Te with other machines in the Shard. In some cases if a replica have missed too many updates, it would perform a standard replication as well as replaying the TR Ansaction log before serving queries.

Both Solrcloud and Master-slave take advantage of solrreplication, but Solrcloud automatically handles rerouting and Recov Ery when all node in the cluster goes down, whereas Master-slave requires some manual work in the event a master becomes u Nresponsive. As a result, turning a master-slave cluster into a HA solution requires a fair amount of work in scripting and case Checki Ng, to ensure no documents is lost and that queries is accurate. By contrast, Solrcloud would never lose updates and would automatically route around any unresponsive nodes.

See also:https://support.lucidworks.com/entries/22180608-solr-ha-dr-overview-3-x-and-4-0-solrcloud-

-------------------------------

Https://support.lucidworks.com/hc/en-us/sections/200314327-Lucene-Solr

http://www.solr.cc/blog/?p=99

What is Solrcloud? (And how does it compare to Master-slave?)