go-mysql-elasticsearch實現mysql 與elasticsearch即時同步深入詳解

這是一個建立於 的文章，其中的資訊可能已經有所發展或是發生改變。

引言：

go-mysql-elasticsearch 是國內作者開發的一款外掛程式。測試表明：該外掛程式優點：能實現同步增、刪、改、查操作。不足之處（待完善的地方）：
1、仍處理開發、相對不穩定階段；
2、沒有日誌，不便於排查問題及查看同步結果。
本文深入詳解了外掛程式的安裝、使用、增刪改查同步測試。

1. go-mysql-elasticsearch 外掛程式安裝

步驟1：安裝go

yum install go

步驟2：安裝godep

go get github.com/tools/godep

步驟3：擷取go-mysql-elastisearch外掛程式

go get github.com/siddontang/go-mysql-elasticsearch

步驟4：安裝go-mysql-elastisearch外掛程式

cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch
make

2.go-mysql-elasticsearch 外掛程式使用

2.1修改設定檔

[root@5b9dbaaa148a etc]# cat river.toml# MySQL address, user and password# user must have replication privilege in MySQL.my_addr = "192.168.1.1:3306"my_user = "root"my_pass = "password@!"# Elasticsearch addresses_addr = "192.168.1.1:9200"# Path to store data, like master.info, and dump MySQL data data_dir = "./var"# Inner Http status addressstat_addr = "192.168.1.1:12800"# pseudo server id like a slave server_id = 1# mysql or mariadbflavor = "mysql"# mysqldump execution pathmysqldump = "mysqldump"# MySQL data source[[source]]schema = "test"# Only below tables will be synced into Elasticsearch.# "test_river_[0-9]{4}" is a wildcard table format, you can use it if you have many sub tables, like table_0000 - table_1023# I don't think it is necessary to sync all tables in a database.tables = ["cc"]# Below is for special rule mapping#[[rule]]#schema = "test"#table = "cc"#index = "go_river"#type = "go_rivert"    # title is MySQL test_river field name, es_title is the customized name in Elasticsearch #   [rule.field]    # This will map column title to elastic search my_title  #  title="es_title"    # This will map column tags to elastic search my_tags and use array type   # tags="my_tags,list"    # This will map column keywords to elastic search keywords and use array type    #keywords=",list"# wildcard table rule, the wildcard table must be in source tables [[rule]]schema = "test"table = "cc"index = "gocc"type = "gocc_t"    # title is MySQL test_river field name, es_title is the customized name in Elasticsearch    [[rule.fields]]    mysql = "mysql101"    elastic = "es_mysql101"

2.2執行同步操作

cd $GOPATH/src/github.com/siddontang/go-mysql-elasticsearch
./bin/go-mysql-elasticsearch -config=./etc/river.toml

3. go-mysql-elasticsearch 外掛程式同步測試結果

3.1插入Insert操作即時同步驗證（驗證ok）

3.1.1Mysql端插入操作

mysql> insert into cc(id,name) values(12, ‘test12’);
Query OK, 1 row affected (0.06 sec)

3.1.2Mysql執行insert後查詢結果

mysql> select * from cc where id =12;
+—-+——–+——–+———————+
| id | name | status | modified_at |
+—-+——–+——–+———————+
| 12 | test12 | ok | 2016-06-24 02:27:29 |
+—-+——–+——–+———————+
1 row in set (0.02 sec)

3.1.3ES端能查詢到新增的value欄位。

[root@5b9dbaaa148a bin]# curl -XGET http://192.168.1.1:9200/gocc/_search?pretty -d '> {"query":> {"term":> {"id":12}}}'{  "took" : 402,  "timed_out" : false,  "_shards" : {    "total" : 8,    "successful" : 8,    "failed" : 0  },  "hits" : {    "total" : 1,    "max_score" : 1.0,    "hits" : [ {      "_index" : "gocc",      "_type" : "gocc_t",      "_id" : "12",      "_score" : 1.0,      "_source" : {        "id" : 12,        "modified_at" : "2016-06-24T02:27:29+01:00",        "name" : "test12",        "status" : "ok"      }    } ]  }}

3.2修改Update操作即時同步驗證（驗證ok）

3.2.1mysql執行更新操作

mysql> update cc set name = 'test12_001' where id = 12;Query OK, 1 row affected (0.05 sec)Rows matched: 1  Changed: 1  Warnings: 0

3.2.2mysql執行修改後查詢

Mysql查詢修改後結果：

mysql> select * from cc where id = 12;+----+------------+--------+---------------------+| id | name       | status | modified_at         |+----+------------+--------+---------------------+| 12 | test12_001 | ok     | 2016-06-24 02:27:29 |+----+------------+--------+---------------------+1 row in set (0.00 sec)

3.2.3 ES查詢修改結果

[root@5b9dbaaa148a bin]# curl -XGET http://192.168.1.1:9200/gocc/_search?pretty -d '{"query":{"term":{"id":12}}}'{  "took" : 59,  "timed_out" : false,  "_shards" : {    "total" : 8,    "successful" : 8,    "failed" : 0  },  "hits" : {    "total" : 1,    "max_score" : 1.0,    "hits" : [ {      "_index" : "gocc",      "_type" : "gocc_t",      "_id" : "12",      "_score" : 1.0,      "_source" : {        "id" : 12,        "modified_at" : "2016-06-24T02:27:29+01:00",        "name" : "test12_001",        "status" : "ok"      }    } ]  }}

3.3刪除操作即時同步驗證

3.3.1Mysql執行刪除操作

mysql> delete from cc where id = 12;Query OK, 1 row affected (0.04 sec)

3.3.2刪除後查詢表

mysql> select * from cc;+----+--------------------+--------+---------------------+| id | name               | status | modified_at         |+----+--------------------+--------+---------------------+|  1 | laoyang360         | ok     | 0000-00-00 00:00:00 ||  2 | test002            | ok     | 2016-06-23 06:16:42 ||  3 | dlllaoyang360      | ok     | 0000-00-00 00:00:00 || 11 | test11             | ok     | 2016-06-24 02:09:15 ||  5 | jdbc_test_update08 | ok     | 0000-00-00 00:00:00 ||  7 | test7              | ok     | 0000-00-00 00:00:00 ||  8 | test008            | ok     | 0000-00-00 00:00:00 ||  9 | test009            | ok     | 0000-00-00 00:00:00 || 10 | test10             | ok     | 2016-06-24 02:08:14 |+----+--------------------+--------+---------------------+9 rows in set (0.02 sec)

3.3.3ES查詢刪除後結果

[root@5b9dbaaa148a bin]# curl -XGET http://192.168.1.1:9200/gocc/_search?pretty -d '{"query":{"term":{"id":12}}}'{  "took" : 40,  "timed_out" : false,  "_shards" : {    "total" : 8,    "successful" : 8,    "failed" : 0  },  "hits" : {    "total" : 0,    "max_score" : null,    "hits" : [ ]  }}

4小結

驗證發現：
（1）go-mysql-elasticsearch外掛程式可以實現同步insert、update、delete操作。
（2）可視化做的不好，沒有列印日誌。
（3）go-mysql-elasticsearch尚不大穩定，出現過無法同步成功的情況，但沒有報錯。不便於排查。

作者：銘毅天下
轉載請標明出處，原文地址：http://blog.csdn.net/laoyang360/article/details/51771483
如果感覺本文對您有協助，請點擊‘頂’支援一下，您的支援是我堅持寫作最大的動力，謝謝！