Elasticsearch Performance Optimization Scheme

Elasticsearch performance optimization is mainly divided into 4 aspects.

First, Server Deployment

1. Add 1-2 servers for load Balancing section Point There are 2 parameters in the Elasticsearch configuration file: Node.master and Node.data.  When used together, these two parameters can help provide server performance.

Node.master:false

Node.data:true

the node server serves only as a data node and is used to store index data only .     Make the node server function single, only for data storage and data query, reduce its resource consumption rate.

Node.master:true

Node.data:false

the node server acts as a master node only, but does not store any index data. The node server will use the

Their own free resources to coordinate the various creation of index requests or query requests that are reasonably distributed to the relevant node servers.

Node.master:false

Node.data:false

The node server will not be selected as the primary node and will not store any index data.

This server is primarily used for query load balancing. When querying, it usually involves querying data from multiple node servers, and requesting distribution to multiple named node servers, and a summary processing of the results returned by each node server, which is eventually returned to the client.

Note: The server can be configured according to the above conditions

2. in the production environment as far as possible to turn off HTTP functionality in the Data node server , also do not install head, Bigdesk, Marvel and other monitoring plug-ins, so that the Data node server can only handle the creation/update/delete/query index data and other operations. HTTP features can be opened on non-data node servers, and the related monitoring plug-ins are also installed on these servers to monitor data such as Elasticsearch cluster status . This is done for data security considerations and, secondly, for service performance considerations.

Parameter setting: Http.enabled:false

3, it is best to deploy only one node on a server , A physical server can start multiple node server nodes (by setting different boot ports), but the CPU, memory, and hard disk resources on a single server are limited after all, from server performance considerations, in the production environment   It is not recommended to start multiple node nodes on a single server.

Second, the server configuration

1. Configure the size of the index thread pool The Elastisearch server has multiple thread pool size configurations.   Mainly include: Index,search,suggest,get,bulk,percolate,snapshot,snapshot_data,warmer,refresh.

This is primarily a configuration adjustment for index and search.

Index : Create/Update/delete index data.

Search :  mainly for the user's various search operations.

The specific configuration is as follows:

ThreadPool

Index

Type:fixed

size:24 ( Logical Core Number )

Queue_ size:1000

Search

Type:fixed

size:24 ( Logical Core Number )

Queue_ size:1000

2. determine the number of shards (shard) and the number of replicas (replica) Elasticsearch when creating index data, it is best to specify the associated shards quantity and replicas, otherwise the server   The default configuration parameter is Shards=5,replicas=1. Because the settings of these two properties directly affect the execution of indexes and search operations in the cluster. Assuming you have enough machines to hold fragments and replicas, you can set these two values as follows:

1) having more fragments can improve the index execution capability and allow a large index to be distributed through the machine ;

2) Having more replicas can improve search execution and clustering capabilities. For an index, number_of_shards can only be set once, and Number_of_replicas may be incremented or reduced at any time using the index Update Settings API. These two configuration parameters are configured in the configuration file as follows:

index.number_of_shards: 5 (Value of First shard)   

index.number_of_replicas:1 ( variable )      

3.elasticsearch.yml Log Logs its specific configuration is as follows:

Index.search.slowlog.level:INFO

Index.search.slowlog.threshold.query.warn:5s

Index.search.slowlog.threshold.query.info:2s

index.search.slowlog.threshold.query.debug:500s

Index.search.slowlog.threshold.query.trace:500ms

Index.search.slowlog.threshold.fetch.warn:5s

Index.search.slowlog.threshold.fetch.info:2s

Index.search.slowlog.threshold.fetch.debug:500ms

Index.search.slowlog.threshold.fetch.trace:200ms

Elasticsearch Performance Optimization Scheme