elasticsearch5.2.1 synchronizing MySQL with Logstash

CentOS self-test can be installed first mysql,elasticsearch do not understand, please refer to another article

Installing Logstash
Official: https://www.elastic.co/guide/en/logstash/current/installing-logstash.html
1. Download the public key
RPM--import Https://artifacts.elastic.co/GPG-KEY-elasticsearch
2. Add Yum Source
Vim/etc/yum.repos.d/logstash.repo
Write in File
[logstash-5.x]
Name=elastic repository for 5.x packages
Baseurl=https://artifacts.elastic.co/packages/5.x/yum
Gpgcheck=1
Gpgkey=https://artifacts.elastic.co/gpg-key-elasticsearch
Enabled=1
Autorefresh=1
Type=rpm-md
Save exit
3. Installing with Yum
Yum Install Logstash
4. Verify that the installation is successful
Enter the Logstash installation directory
Cd/usr/share/logstash
Run
Bin/logstash-e ' input {stdin {}} ' output {stdout {}} '
Wait a few seconds to appear
The stdin plugin is now waiting for input:
And then enter
Hello World
Get a similar result
2016-11-24t08:01:55.949z Bogon Hello World

Installing the LOGSTASH-INPUT-JDBC Plugin

1. Install Ruby and RubyGems (note: Ruby is required in version 1.8.7 or more)
# yum Install-y Ruby RubyGems
To check the ruby version:
# ruby-v
Ruby 1.8.7 (2013-06-27 patchlevel 374) [X86_64-linux]
Replace mirrors in the country
Gem Sources--remove http://rubygems.org/

Gem Sources-a http://gems.ruby-china.org/
Verify success
Gem Sources-l
Modify the Gemfile data source address
Whereis Logstash # Check the location of the Logstash installation, my/usr/share/logstash directory

Cd/usr/share/logstash
Vim Gemfile
Modify the value of source to: "https://gems.ruby-china.org/"

Vim Gemfile.jruby-1.9.lock

# Locate remote to modify its value to: https://gems.ruby-china.org/

Start installation

./bin/logstash-plugin Install--no-verify LOGSTASH-INPUT-JDBC

2. Write the config file to start synchronizing VI xxx.conf

[HTML]View PlainCopy

2. Input {
4. JDBC {
6. type => "User"
8. jdbc_connection_string => "jdbc:mysql://192.168.33.101:3306/test"
10. Jdbc_user => "root"
12. Jdbc_password => "123456"
14. jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.41.jar"
16. Jdbc_driver_class => "com.mysql.jdbc.Driver"
18. jdbc_paging_enabled => "true"
20. jdbc_page_size => "+"
22. statement => "SELECT * from user"
24. schedule => "* * * * * *"
26. }
30. JDBC {
32. type => "task"
34. jdbc_connection_string => "jdbc:mysql://192.168.33.101:3306/test"
36. Jdbc_user => "root"
38. Jdbc_password => "123456"
40. jdbc_driver_library => "/usr/share/logstash/mysql-connector-java-5.1.41.jar"
42. Jdbc_driver_class => "com.mysql.jdbc.Driver"
44. jdbc_paging_enabled => "true"
46. jdbc_page_size => "+"
48. statement => "SELECT * from Task"
50. schedule => "* * * * * *"
52. }
54. }
58. filter{
60. mutate{
62. Remove_field => ["@timestamp", "@version"]
64. }
66. }
70. Output {
72. if [type] = = "User" {
74. Elasticsearch {
76. hosts => ["192.168.33.111:9200", "192.168.33.112:9200", "192.168.33.113:9200"]
78. index => "Testindex"
80. Document_type => "User"
82. document_id => "%{id}"
84. user => "Elastic"
86. password => "changeme"
88. }
90. }
92. if [type] = = "task" {
94. Elasticsearch {
96. hosts => ["192.168.33.111:9200", "192.168.33.112:9200", "192.168.33.113:9200"]
98. index => "Testindex"
100. Document_type => "task"
102. document_id => "%{id}"
104. user => "Elastic"
106. password => "changeme"
108. }
110. }
112. }

Execution: Bin/logstash-f Xxx.confok is done!!!

elasticsearch5.2.1 synchronizing MySQL with Logstash