elasticsearch2.3/2.4 Upgrade to ElasticSearch5.0

elasticsearch2.3/2.4 Upgrade to ElasticSearch5.0

Reference documents (in the order of precedence)

Https://www.elastic.co/guide/en/elasticsearch/reference/5.0/setup-upgrade.html

https://github.com/elastic/elasticsearch-migration/

Http://www.infoq.com/cn/news/2016/08/Elasticsearch-5-0-Elastic

http://hnr520.blog.51cto.com/4484939/1867033

Note It's best to run with elasticsaerch-migration. Note, I am here to hold the goal of losing data index to upgrade.

Recently, elastic in the official blog announced that Elasticsearch 5.0 officially released. This version is based on Lucene 6.2.0 and has been deployed on elastic cloud. It is said to be the fastest, safest, and most easy-to-use version yet.

Elasticsearch 5.0 brings many enhancements and new features, mainly including:

Index performance: Thanks to several improvements, including better numeric data structures, index throughput has increased significantly. The index throughput is increased from 25% to 80% depending on the application scenario.

Ingest node: Adding data to Elasticsearch is easier. Logstash is a powerful tool, while some smaller users simply need a filter and do not require many of the routing options it offers. As a result, elastic has implemented some of the most popular Logstash filters (such as Grok, split) directly in Elasticsearch as processors. Multiple processors can be combined into a single pipe that is applied to the document at index time.

Painless script: Scripts are used in many places in Elasticsearch, and scripts are disabled by default for security reasons, which is quite disappointing. To this end, elastic has developed a new scripting language painless. The language is faster, more secure, and is enabled by default. Not only that, it executes 4 times times more than groovy, and it's getting faster. Painless has become the default scripting language, and groovy, JavaScript, and Python have been deprecated. To learn more about this new language, please click here.

New data structure: Lucene 6 brings a new points data structure k-d tree that stores numeric and geo-location fields, radically altering the index and search methods of numeric values. Benchmark tests showed that points increased the query speed by 36%, increased the index speed by 71%, and consumed disk and memory space by 66% and 85%, respectively (see "Searching for values in 5.0").

Search and Aggregation: Kibana chart generation is significantly faster with instant aggregation. Elastic has been refactoring the search API for a year, and Elasticsearch can now perform more intelligently with scoping queries, recalculating aggregations only for indexes that have changed, instead of recalculating each query from the beginning. In terms of search, the default correlation calculation has been replaced by TF/IDF with more advanced BM25. The completion recommendation program has been completely rewritten to take the deleted document into account as well.

More friendly: Elasticsearch 5.0 is safer and easier to use. They used the "early tip" approach. If there is a problem, the new version will prompt you early. For example, Elasticsearch 5.0 will rigorously validate the settings. If it does not recognize the value of a setting, hints and suggestions are given. In addition, cluster and index settings can now be lifted by null. In addition, there are other improvements, such as the rollover and shrink APIs, which enable a new mode to manage time-based indexes, introduce new Cluster-allocation-explain APIs, and simplify index creation.

Elasticity: Elasticsearch Each part of the distributed model is decomposed, reconstructed, and simplified to improve reliability. The cluster status update will now wait for all nodes in the cluster to confirm. If a copy slice (replica shard) is marked as failed by the Master slice (primary), the master will wait for the primary node (master) response. The index now uses the UUID in the data path, not the index name, to avoid naming conflicts. In addition, Elasticsearch now initiates a check to ensure that the system configuration is not a problem. Configuration is cumbersome, but if it's just a trial, developers can choose Localhost-only mode to avoid tedious configuration. In addition, the new version also adds circuit breakers and other soft limits, limiting the amount of memory requested to use, and protecting the cluster from attacks by malicious users.

In addition, the release provides a low-level Java Rest/http client that can be used for listening, logging, request polling, failed node retries, and so on. It uses Java 7, minimizing dependencies and less dependency conflicts than transport clients. In the benchmark, however, its performance is not lost to the transport client. However, this is a bottom-level client and no Query Builder or helper is currently available. Its input parameters and output results are JSON.

It is important to note that this release introduces a number of disruptive changes, but fortunately they provide a migration helper to help developers migrate from Elasticsearch 2.3.x/2.4.x to Elasticsearch 5.0. If you are migrating from an earlier Elasticsearch version to the latest version 5.0, consult the upgrade documentation.

1, uninstall the original software to install new software

Cd/usr/local/src

Rpm-e Elasticsearch

Rpm-e Kibana

RPM-IVH elasticsearch-5.0.0.rpm kibana-5.0.0-x86_64.rpm

2, probably looked under the configuration file no changes to directly with the original

Diff Elasticsearch.yml Elasticsearch.yml.rpmsave

MV Elasticsearch.yml Elasticsearch.yml.default

Cp-f Elasticsearch.yml.rpmsave elasticsearch.yml

Chown elasticsearch:elasticsearch/etc/elasticsearch/elasticsearch.yml

3, according to the official website recommended to add the following configuration to the ELASTICSEARC.YML master configuration file

Http.cors.enabled:true

Http.cors.allow-origin: "*"

4, start a new Elasticsearch

The startup process has failed, and the following issues are summarized by looking at the error log:

Question One

[2016-11-07t11:37:54,191] [WARN] [O.e.b.jnanatives] Unable to lock JVM memory:error=12, Reason=cannot allocate Memory

[2016-11-07t11:37:54,191] [WARN] [O.e.b.jnanatives] This can result in part of the JVM being swapped out.

[2016-11-07t11:37:54,191] [WARN] [o.e.b.jnanatives] Increase rlimit_memlock, soft limit:65536, hard limit:65536

[2016-11-07t11:37:54,191] [WARN] [O.e.b.jnanatives] These can is adjusted by modifying/etc/security/limits.conf, for example:

# Allow user ' Elasticsearch ' mlockall

Elasticsearch Soft Memlock Unlimited

Elasticsearch Hard Memlock Unlimited

Workaround:

/etc/security/limits.conf

Elasticsearch Soft Memlock Unlimited

Elasticsearch Hard Memlock Unlimited

Question Two

Found index level settings at node level configuration.

Since Elasticsearch 5.x Index level settings can not is set on the nodes

Configuration like the Elasticsearch.yaml, in System Properties or command line

Arguments. In order to upgrade all indices the settings must be updated via the

/${index}/_settings API. Unless all settings is dynamic all indices must be closed

In order to apply the upgradeindices created in the future should use index templates

To set default values.

Workaround:

Note In the main configuration file, the official recommendation is to create an index and then set

# Index.number_of_shards:5

# index.number_of_replicas:1

Ensure all required values is updated on all indices by executing:

Curl-xput ' http://localhost:9200/_all/_settings?preserve_existing=true '-d ' {

"Index.number_of_replicas": "1",

"Index.number_of_shards": "10"

}‘

Question Three

Java.lang.IllegalArgumentException:unknown setting [Node.rack] Please check the any required plugins is installed, or C Heck the breaking changes documentation for removed settings

Workaround:

This is the only difference in the diff place, as to why this change to the official website has introduced

# NODE.RACK:R1

Node.attr.rack:r1

Question Four

[2016-11-07t12:26:05,957] [ERROR] [O.e.b.bootstrap] Exception

Java.lang.IllegalArgumentException:unknown setting [script.indexed] did you mean any of [Script.inline, Script.ingest]?

Workaround:

At present I commented, this should be a grammatical error, follow-up re-check the official website

#script. inline:true

#script. indexed:true

Question Five,

Max file descriptors [65535] for elasticsearch process likely too low, increase to at least [65536]

Max number of threads [1024x768] for user [elasticsearch] likely too low, increase to at least [2048]

Workaround:

The error message directly prompts to modify the relevant parameters of the LIMITS.CONF

Elasticsearch Soft Nofile 65536

Elasticsearch Hard Nofile 131072

Elasticsearch Soft Nproc 2048

Elasticsearch Hard Nproc 4096

This article is from the "jerrymin" blog, make sure to keep this source http://jerrymin.blog.51cto.com/3002256/1870205

elasticsearch2.3/2.4 Upgrade to ElasticSearch5.0