Elk6+filebeat+kafka installation Configuration

Source: Internet
Author: User
Tags zookeeper kibana logstash filebeat

1, installation Elasticsearch
1.) Turn off the firewall and SELinux
Service Iptables Stop
Chkconfig iptables off
Chkconfig iptables--list
Vim/etc/sysconfig/selinux
Selinux=disabled
Setenforce 0

2.) Configuring the JDK environment
vim/etc/profile.d/java.sh
Export java_home=/home/admin/jdk1.8.0_172/
Export classpath=.: $JAVA _home/lib.tools.jar
Export path= $JAVA _home/bin: $PATH
source/etc/profile.d/java.sh

3.) Install elasticsearch6.x
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
TAR-ZXVF elasticsearch-6.2.4.tar.gz-c/home/admin/project/elk
cd/home/admin/project/elkelasticsearch-6.2.4
Vim Config/elasticsearch.yml
Cluster.name:elasticsearch
Node.name:node-1
network.host:10.2.151.203
http.port:9200
Bootstrap.memory_lock:false
Bootstrap.system_call_filter:false
Http.cors.enabled:true
Http.cors.allow-origin: "*"

4.) Start Elasticsearch
Useradd Elk
Chown–r elk.elk/home/admin/project/elk/elasticsearch-6.2.4
./bin/elasticsearch–d

NETSTAT–LUNTP #查看监听端口9200 9300

Curl 10.2.151.203:9200

5.) Start a common error
Uncaught exception in thread [main]
Org.elasticsearch.bootstrap.StartupException:java.lang.RuntimeException:can not run Elasticsearch as root
Problem reason: Cannot start with root user
WORKAROUND: Switch to another user to start

Unable to install Syscall filter:
Java.lang.UnsupportedOperationException:seccomp Unavailable:
Cause: It's just a warning, mainly because your Linux version is too low to cause
WORKAROUND: The warning does not affect use and can be ignored

Error:bootstrap checks failed
Memory locking requested for Elasticsearch process but memory was not locked
Problem reason: Failed to lock memory
WORKAROUND: Switch to the root user and edit the limits.conf configuration file
Vim/etc/security/limits.conf

    • Hard Nproc 65536
    • Soft Nproc 65536
    • Hard Nofile 65536
    • Soft Nofile 65536

Max number of threads [1024x768] for user [es] was too low, increase to at least [2048]
Cause: Unable to create local thread problem, user maximum number of threads to create is too small
Solution: Switch to the root user, enter the LIMITS.D directory, modify the 90-nproc.conf configuration file
Vim/etc/security/limits.d/90-nproc.conf

    • Soft Nofile 65536
    • Soft Nproc 65536
    • Soft Nproc 2048

Max virtual Memory Areas Vm.max_map_count [65530] is too low, increase to at least [262144]
Cause: The maximum virtual memory is too small
Solution: Switch to the root user and modify the configuration file sysctl.conf
Vim/etc/sysctl.conf
vm.max_map_count=655360
Sysctl-p

System call filters failed to install; Check the logs and fix your configuration or disable system call filters at your own risk
Cause of the problem: because CENTOS6 does not support Seccomp
Workaround: Configure Bootstrap.system_call_filter to False in Elasticsearch.yml, and note the following in memory:
Bootstrap.memory_lock:false
Bootstrap.system_call_filter:false

2, Elasticsearch-head plug-in installation
View Elasticsearch cluster status information through the Web interface

1.) Download and install Nodejs
wget HTTPS://NODEJS.ORG/DIST/V8.11.3/NODE-V8.11.3-LINUX-X64.TAR.XZ
TAR-ZXVF node-v8.11.3-linux-x64.tar.gz-c/home/admin/project/elk/
cd/home/admin/project/elk/
MV node-v8.11.3-linux-x64/node-v8.11.3
#配置nodejs环境变量
vim/etc/profile.d/node.sh
Export node_home=/home/admin/project/elk/node-v8.11.3
Export path= $NODE _home/bin: $PATH
Export Node_path= $NODE _home/lib/node_modules
source/etc/profile.d/node.sh
#查看nodejs是否生效
[Email protected] node-v8.11.3]$ node-v
v8.11.3
[Email protected] node-v8.11.3]$ npm-v
5.6.0

2.) Install Grunt
NPM Config Set Registry https://registry.npm.taobao.org
Vim ~/.NPMRC
registry=https://registry.npm.taobao.org
Strict-ssl = False
NPM install-g GRUNT-CLI
#将grunt加入系统文件
Ln-s/home/admin/project/elk/node-v8.11.3/lib/node_modules/grunt-cli/bin/grunt/usr/bin/grunt

3.) Download Head binary package
wget Https://codeload.github.com/mobz/elasticsearch-head/zip/master
Unzip Elasticsearch-head-master.zip
CD Elasticsearch-head-master
NPM Install
#如果速度较慢或安装失败, we recommend the use of domestic mirrors
NPM Install--ignore-scripts-g cnpm--registry=https://registry.npm.taobao.org

4.) Modify the Elasticserach configuration file
VI./CONFIG/ELASTICSEARCH.YML
#增加新的参数 so that the head plugin can access ES
Http.cors.enabled:true
Http.cors.allow-origin: "*"

5.) Modify the Gruntfile.js configuration
Vim Gruntfile.js
#port: 9100 add hostname address above
Hostname: "0.0.0.0",

6.) Modify the _site/app.js configuration
Vim _site/app.js
#localhost替换为IP地址
This.base_uri = This.config.base_uri | | This.prefs.get ("App-base_uri") | | "Http://10.2.151.203:9200";

7.) Start Grunt
Grunt Server
#如果启动成功, you can run directly in the background and the command line can continue typing (but if you want to quit, you need to kill the process yourself)
Grunt Server &
Nohup Grunt Server & Exit #后台启动

#启动提示模块未找到

> Local Npm Module "Grunt-contrib-jasmine" not found. Is it installed?
NPM Install Grunt-contrib-jasmine #安装模块

3, Installation Kibana
1.) Download and install
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz
TAR-ZXVF kibana-6.2.4-linux-x86_64.tar.gz-c/home/admin/project/elk/
Cd/ho me/admin/project/elk/kibana-6.2.4-linux-x86_64
2.) Modify the configuration
Vim Config/kibana.yml
server.port:5601
Server.host: "IP"
elasticsearch.url:http://ip:9200

3.) Start Kibana
./bin/kibana

4, Installation Logstash
1.) Download and install
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz
TAR-ZXVF logstash-6.2.4.tar.gz-c/home/admin/project/elk/
cd/home/admin/project/elk/logstash-6.2.4
2.) Create a new template
Vim config/test.conf
Input
{
Kafka
{
Bootstrap_servers = "10.7.1.112:9092"
topics = "Nethospital_2"
codec = "JSON"
}
}

Output
{
if [fields][tag] = = "Nethospital_2"
{
Elasticsearch
{
hosts = ["10.7.1.111:9200"]
index = "Nethospital_2-%{+yyyy-mm-dd}"
codec = "JSON"
}
}
}

3.) Start Logstash
Nohup./bin/logstash–f config/test.conf & #-F Specify configuration file
5, Installation Kafka
1.) Download and install
wget https://archive.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz
wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.12/zookeeper-3.4.12.tar.gz
TAR-ZXVF kafka_2.11-1.0.0.tgz-c/home/admin/project/elk/
TAR-ZXVF zookeeper-3.4.12.tar.gz-c/home/admin/project/elk/
cd/home/admin/project/elk/kafka_2.11-1.0.0/

2.) Modify the Kafka parameters and start
Vim Config/zookeeper.properties
Datadir=/tmp/zookeeper/data # Data Persistence path
clientport=2181 # Connection Port
MAXCLIENTCNXNS=100 # Maximum number of connections
Datalogdir=/tmp/zookeeper/logs #日志存放路径
ticktime=2000 # Zookeeper Server heartbeat time, per millisecond
Initlimit=10 # votes to elect the initial time of the new leader.
#启动zookeeper
./bin/zookeeper-server-start.sh Config/zookeeper.properties
#后台启动
Nohup./bin/zookeeper-server-start.sh Config/zookeeper.properties &

3.) Modify the Kafka parameters and start
Vim Config/server.properties
Broker.id=0
port=9092
host.name=10.2.151.203
Num.network.threads=3
Num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
Log.dirs=/data/logs/kafka
num.partitions=2
Num.recovery.threads.per.data.dir=1
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

#启动zookeeper
./bin/kafka-server-start.sh Config/server.properties
#后台启动
Nohup bin/kafka-server-start.sh Config/server.properties &

4.) Test Kafka
#创建topic (Test)
bin/kafka-topics.sh--create--zookeeper 10.2.151.203:2181--replication-factor 1--partitions 1--topic test

#查看topic
bin/kafka-topics.sh--list--zookeeper 10.2.151.203:2181

#启动生产进程测试
bin/kafka-console-producer.sh--broker-list 10.2.151.203:9092--topic test

#启动启动消费者进程
bin/kafka-console-consumer.sh--zookeeper 10.2.151.203:2181--topic test--from-beginning

6, Installation Filebeat
1.) Download and install
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz
TAR–ZXVF Filebeat-6.2.4-linux-x86_64.tar.gz–c/home/admin/project/elk
Cd/home/admin/project/elk/filebeat-6.2.4-linux-x86_64

2.) Configure Filebeat
Vim Filebeat.yml

    • Input_type:log
      #Paths that should is crawled and fetched. Glob based paths.
      Paths
    • /home/admin/project/other_project/nh-interface/nh-interface.log
      Fields
      Tag:nh-interface
      Multiline
      Pattern: ' ^[0-9]{4}-[0-9]{2}.* '
      Negate:true
      Match:after

Output.kafka:
Enabled:true
Hosts: ["appelk1:9092", "appelk2:9092", "appelk3:9092"]
Topic: ' Hospital '
Compression:gzip
max_message_bytes:100000000

3) Start Filebeat
Nohup./FILEBEAT-E-C Filebeat.yml &

View cluster status
Curl-xget ' Http://10.2.151.203:9200/_cat/nodes '
Curl-xget ' Http://10.2.151.203:9200/_cat/nodes?v '
Curl-xget ' Http://10.2.151.203:9200/_cluster/state/nodes?pretty '

View Cluster Master
Curl-xget ' Http://10.2.151.203:9200/_cluster/state/master_node?pretty '
or Curl-xget ' http://10.2.151.203:9200/_cat/master?v '

Querying the health status of a cluster
Curl-xget ' Http://10.2.151.203:9200/_cluster/health?pretty '
Or
Curl-xget ' Http://10.2.151.203:9200/_cat/health?v '

7. Install Cerebro plug-in
Cerebo is a Kopf on ES5 that manages and monitors elasticsearch cluster state information through a web interface

1.) Download and install
#wget https://github.com/lmenezes/cerebro/releases/download/v0.8.1/cerebro-0.8.1.tgz
#tar –ZXVF Cerebro-0.8.1.tgz/home/admin/project/elk
#cd/home/admin/project/elk/cerebro-0.8.1
# #vim Conf/application.conf
#hosts = [* * * *
{
Host = "http://10.2.151.203:9200"
Name = "My-elk"
},
]

2.) Start/access
Nohup./bin/cerebro & #后台运行
http://10.2.151.203:9000

8. Install Bigdesk plug-in
Bigdesk statistical analysis and charting of Elasticsearch cluster status information
1.) Download and install
#wget Https://codeload.github.com/hlstudio/bigdesk/zip/master
Unzip Bigdesk-master.zip
#mv bigdesk-master/home/admin/project/elk/elasticsearch-6.2.4/plugins/
#cd/home/admin/project/elk/elasticsearch-6.2.4/plugins/bigdesk-master/_site**

2. ) use Python-m simplehttpserver to quickly build HTTP services
Specify Port 8000
Nohup python-m simplehttpserver 8000 & #后台运行

Http://10.2.151.203:8000/

Elk6+filebeat+kafka installation Configuration

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.