Simple use of distributed search engine Elasticsearch

Simple use of distributed search engine Elasticsearch

Https://www.elastic.co/products/elasticsearch/

I. Features

1. Supports Chinese Word Segmentation

2. Supports full-text search engines for multiple data sources

3. Distributed

4. lucene-based open-source search engine

5. Restful api

Ii. Resources

• Smartcn, default Chinese Word Segmentation: https://github.com/elasticsearch/elasticsearch-analysis-smartcn

• Mmseg: https://github.com/medcl/elasticsearch-analysis-mmseg

• Ik: https://github.com/medcl/elasticsearch-analysis-ik

• Pinyin, pinyin word segmentation can be used to input pinyin tips Chinese: https://github.com/medcl/elasticsearch-analysis-pinyin

• Stconvert, Chinese Simplified and Traditional Interchange: https://github.com/medcl/elasticsearch-analysis-stconvert

• Elasticsearch-servicewrapper: https://github.com/elasticsearch/elasticsearch-servicewrapper

• Elastic HQ, elasticsearch monitoring tool: http://www.elastichq.org

• Elasticsearch-Rich Text: https://github.com/medcl/elasticsearch-rtf

Iii. Installation

• Server: Linux (centos 6.x)

• Java environment: JDK 1.8.0

• Elasticsearch: 2.3.1

• Elasticsearch-jdbc (Data Source Plug-in): 2.3.1

• IK Analysis (Chinese Word Segmentation plug-in): 1.9.1

1. install Java

yum install java-1.8.0

2. Install Elasticsearch

# Create. repo file (elasticsearch. repo) cat>/etc/yum. repos. d/elasticsearch. repo <EOF [elasticsearch-2.x] name = Elasticsearch repository for 2.x packagesbaseurl = plugin -- import https://packages.elastic.co/GPG-KEY-elasticsearchyum install elasticsearch

3. Create a user

If you start with the root account, the following error is reported:

Exception in thread "main" java.lang.RuntimeException: don't run elasticsearch as root. at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:93) at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:144) at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:285) at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35) Refer to the log for complete error details.

This is a condition for system security considerations. Because ElasticSearch can receive and execute user input scripts, we recommend that you create a separate user to run ElasticSearch for system security considerations.

groupadd elsearchuseradd elsearch -g elsearch -p elasticsearch

4. Create a directory

mkdir -p  /data/elasticsearch/datamkdir -p  /data/elasticsearch/logschown -R elsearch:elsearch /data/elasticsearch/datachown -R elsearch:elsearch /data/elasticsearch/logs

5. Generate the configuration file (/etc/elasticsearch. yml)

# Cluster name (the same cluster must have the same name. name: my-application # service node name (different from each service node) node. name: node-1 # data storage path. data:/data/elasticsearch/data # Service Log path. logs:/data/elasticsearch/logs # service IP address network. host: 0.0.0.0 # service port http. port: 9200

Iv. Install IK 1. Install maven

wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repoyum install apache-maven

2. Download the ik source code package

Git clone https://github.com/medcl/elasticsearch-analysis-ik

3. Generate the jar plug-in package

mvn cleanmvn compilemvn packageunzip target/releases/elasticsearch-analysis-ik-*.zipcp -r target/releases/ /usr/share/elasticsearch/plugins/ik

4. Configure Dictionary (ik comes with sogou dictionary)

Configuration:/usr/share/elasticsearch/plugins/ik/config/ik/IKAnalyzer. cfg. xml

<Entry key = "ext_dict"> custom/mydict. dic; custom/single_word_low_freq.dic; custom/sougou. dic </entry>

Copy the jar package toplugins/analysis-ikDirectory, and then copy the extracted ik directory (configuration and dictionary) toconfigDirectory. Then edit the configuration fileelasticsearch.yml, Add a line to the end:

index.analysis.analyzer.ik.type : "ik"

Restartservice elasticsearch restart

Enter the data and create an index.

 

5. elasticsearch-jdbc1, using feeder

wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/2.3.1.0/elasticsearch-jdbc-2.3.1.0-dist.zipunzip elasticsearch-jdbc-2.3.1.0-dist.zip

Edit the data import script import. sh

export JDBC_IMPORTER_HOME=/elasticsearch-jdbc-2.3.1.0bin=$JDBC_IMPORTER_HOME/binlib=$JDBC_IMPORTER_HOME/libecho '{"type" : "jdbc","jdbc": {"url":"jdbc:mysql://127.0.0.1:3306/dbtest","user":"root","password":"123456","sql":"select * from test_tb","index" : "customer","type" : "external"}}' | java \    -cp "${lib}/*" \    -Dlog4j.configurationFile=${bin}/log4j2.xml \    org.xbib.tools.Runner \    org.xbib.tools.JDBCImporter

Test

curl 'localhost:9200/customer/external/_search?pretty&q=*'

2. Using river

# Install the elasticsearchcurl-OL https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.zipcd $ ES_HOMEunzip path/to/elasticsearch-1.4.2.zip # Install the JDBC plug-in. /bin/plugin -- install jdbc -- url javasdrivercurl-o mysql-connector-java-5.1.33.zip-L 'HTTP: // launch mysql-connector-java-5.1.33-bin.jar $ ES_HOME/plugins/jdbc/chmod 644 $ ES_HOME/plugins/jdbc/* # Start elasticsearch. /bin/elasticsearch # Stop rivercurl-XDELETE 'localhost: 9200/_ river/my_jdbc_river /'

JDBC plug-in Parameters

Curl-XPUT 'localhost: 9200/_ river/my_jdbc_river/_ meta'-d' {"type": "jdbc", "jdbc": {"url": "jdbc: mysql: // localhost: 3306/test "," user ":" "," password ":" "," SQL ":" select * from orders "," index ": "myindex", "type": "mytype ",...}} 'If an array is passed to the jdbc field, multiple river sources can also use curl-xput' localhost: 9200/_ river/my_jdbc_river/_ meta '-d' {<river parameters> "type": "jdbc", "jdbc": [{<river definition 1> }, {<river definition 2 >}]} 'curl-xput' localhost: 9200/_ river/my_jdbc_river/_ meta'-d' {"type": "jdbc ", "jdbc": {"driver": "com. mysql. jdbc. driver "," url ":" jdbc: mysql: // localhost: 3306/test "," user ":" root "," password ":" 123456 ", "SQL": "select * from test. student; "," interval ":" 30 "," index ":" test "," type ":" student "}}'

Check whether ES has synchronized the data.

curl -XGET 'localhost:9200/test/student/_search?pretty&q=*'

Address: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

 

Reference

Https://www.elastic.co/guide/en/elasticsearch/guide/current/empty-search.html

Https://github.com/medcl/elasticsearch-analysis-ik

Http://blog.csdn.net/clementad/article/details/46898013

Https://endymecy.gitbooks.io/elasticsearch-guide-chinese/content/elasticsearch-river-jdbc.html

Https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html

Https://github.com/jprante/elasticsearch-jdbc

Http://www.voidcn.com/blog/wojiushiwo987/article/p-6058574.html

Http://leotse90.com/2015/11/11/ElasticSearch%E4%B8%8EMySQL%E6%95%B0%E6%8D% AE %E5%90%8C%E6%AD%A5%E4%BB%A5%E5%8F%8A%E4%BF% AE %E6%94%B9%E8%A1%A8%E7%BB%93%E6%9E%84/

Http://www.jianshu.com/p/638ff7b848cc

Http://www.cnblogs.com/buzzlight/p/logstash_elasticsearch_kibana_log.html