Analysis of distributed ES operation Flow

Conceptual resolution curd Operations

Curd operations are specific to one or more documents, and each document's routing is confirmed, so its shard can be determined beforehand. This process corresponds to the Document API of ES.

• NEW: Refers to the process of indexing a document.
• Retrieval: Refers to the process of getting one or more specific documents from ES.
• Delete: Refers to deleting a document from ES so that it is no longer searchable.
• Update: Refers to the process of updating a document in ES, which is essentially the process of deleting + new.

Search

A search operation is the process of getting a matching document from ES through a query condition, without knowing which document matches the query before searching. This process corresponds to the Search API for ES.

Routing and Shard shards

• When the document is indexed, you need to determine which shard the document is stored on. (by using _id as routing to calculate the Shard)
• When the document is retrieved, you need to determine which shard the document is on. (by using _id as routing to calculate the Shard)

Routing

The determination of shards is accomplished by routing, with the following formula:

Shard = hash (routing)% number_of_primary_shards

• The routing value is an arbitrary string, which is _id by default but can also be customized.
• Routing strings generate a number by a hash function and then divide by the number of primary slices to get a remainder (remainder), the remainder range is always 0 to number_of_primary_shards-1, which is the number that a particular document
  In the Shard.
• This also explains why the number of primary shards can only be defined when the index is created and cannot be modified: if the number of primary shards changes in the future, all previous route values are invalidated and the document is never found.

New, index, and delete processes for documents

New, indexed, and delete requests are write operations that must be completed successfully on the primary shard to replicate to the related replication shards. And wait until all the replication shards are complete before returning to the request node.

The necessary sequential steps to successfully create, index, or delete a document on primary and replicated shards:

• 1. The client sends a new, indexed, or delete request to Node 1.
• 2. Node 1 uses the document's _id to determine that the document belongs to Shard 0 (shard is calculated by using _id as the routing). It forwards the request to Node 3, and the Shard 0 is bit on it.
• 3. Node 3 Executes the request on the primary Shard and, if successful, forwards the request to the corresponding replication node at node 1 and Node 2. When all of the replication nodes report success, node 3 reports success to the requested node nodes 1, and the requested nodes are then reported back to the client.

When the client receives a successful response, the modification of the document has been applied to the primary shard and all replicated shards.

The write operation is time-consuming because both the primary shard and the copy shard are successful before they are returned successfully.

Optimized replication:

Replication default to sync. This means that all replicated shards are returned after they have been manipulated.

Set to Async to run when the primary shard operation is complete returns.

Process of retrieving documents

Retrieves the document as a read operation, and requests that only one copy of the Shard return the result of the operation is complete.

The necessary sequential steps for retrieving a document on a primary shard or a replicated shard:

• 1. The client sends a GET request to the Node1 (Master node).
• 2. The node uses the _id of the document to determine that the document belongs to Shard 0(shard is calculated by using _id as the routing) . The replication shards corresponding to Shard 0 are available on three nodes. At this point, it forwards the request to Node 2.
• 3. Node 2 is returned to Node 1, and Node 1 is returned to the client.

For read requests, in order to balance the load, the request node chooses a different copy of the request for each shard-it loops through all the Shard replicas.

Update process for documents

The overall process of the update process is the "read" + "write" operation.

The necessary sequential steps to perform the update:

• 1. The client sends an update request to Node 1.
• 2. It forwards the request to the node 3 where the primary shard resides.
• 3. Node 3 Retrieves the document from the primary Shard, modifies the JSON for the _source field, and rebuilds the index on the primary shard. If another process modifies the document, it repeats step 3 with the number of times the retry_on_conflict is set, and it is discarded if it is unsuccessful.
• 4. If Node 3 successfully updates the document, it forwards the new version of the document to the Replication node on Node 1 and Node 2 to rebuild the index. When all the replication nodes report success, node 3 returns to the request node successfully and then returns to the client.

Batch document operation flow

The bulk operation process is basically divided into the following steps:

• 1. The request node confirms the target shard for each operation request and re-groups it according to the target Shard requested
• 2. Forward each set of requests to the target primary shard
• 3. Each target primary shard node executes each action individually ( read/write operations are the same as described earlier )
• 4. All nodes return execution results to the request node
• 5. Request the node to organize the results and feedback to the user

Search

A CRUD operation for a document processes only a single document at a time (bulk operations are also performed individually). In a CRUD operation, the uniqueness of the document is determined by the combination of _index, _type, and Routing-value (which is usually the _id of the document by default). This means that we can know exactly which shard in the cluster has this document.

The search process requires a more complex model because it does not know which document will match the query (the document may reside on any shard in the cluster). Search to see if any matching documents are included by querying each shard copy of the index that we are interested in.

The execution of a search is performed in two stages, called a query and then retrieved (query then fetch).

Query phase

GET/_search
{"From": 90,
"Size": 10
}

1. The client sends a search request to node 3, and Node 3 creates a null-priority queue of length from+size.

2. Node 3 Forwards this search request to the original or copy of each shard in the index. Search requests can be processed by the original or any copy of each shard.

(Not all replicas handle the same request, instead polling for different requests, so multiple replicas can increase throughput)

3. Each shard executes the query locally and results in an ordered local priority queue of size from+size.

4. Each shard returns the ID of the document and the sort value of all the document in its priority queue to Node 3 of the Coordinator. Node 3 Merges these values into its own priority queue to produce a global ordering result.

5. The mechanism for searching for multiple (multiple) or full (all) indexes is exactly the same--just a few more shards.

Retrieval phase

The query phase identifies the document that satisfies the search request, but we still need to retrieve the document itself. This is the work of the retrieval phase.

1. The Coordination node (request node) identifies which document needs to be retrieved (for example, only the first 100 items) and makes a GET request to the relevant shard.

2. Each shard loads the document and enriches (enrich) them as needed before returning the document to the Reconcile node.

3. Once all the document has been retrieved, the coordination node returns the result to the client.

Search option 1. Preference

The preference parameter allows you to control which shard or node is used to process search requests. She accepts some of the following parameters _primary, _primary_first, _local, _only_node:xyz, _prefer_node:xyz and _shards:2,3

2. Timeout:

Typically, the coordination node waits for the answer to receive all shards. If one of the nodes encounters a problem, it slows down the entire search request. The timeout parameter tells the coordination node how long to wait and can discard the wait and return the results.

3. Routing:

Specify one or more routing values to limit the search for only those shards, not all of the shards in index.

4. Search_type:

• Count: You can use this query type when you do not need search results to know only the number of document that satisfies the query.
• Query_and_fetch: Search type merges the query and retrieval phases into one step
• Dfs_query_then_fetch and Dfs_query_and_fetch
• The Scan:scan (scan) search type is used in conjunction with the scroll (scrolling) API to efficiently retrieve a large number of results. It is implemented by disabling sorting.

Analysis of distributed ES operation Flow