Elk6+filebeat+kafka installation Configuration

1, installation Elasticsearch
1.) Turn off the firewall and SELinux
Service Iptables Stop
Chkconfig iptables off
Chkconfig iptables--list
Vim/etc/sysconfig/selinux
Selinux=disabled
Setenforce 0

2.) Configuring the JDK environment
vim/etc/profile.d/java.sh
Export java_home=/home/admin/jdk1.8.0_172/
Export classpath=.: $JAVA _home/lib.tools.jar
Export path= $JAVA _home/bin: $PATH
source/etc/profile.d/java.sh

3.) Install elasticsearch6.x
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
TAR-ZXVF elasticsearch-6.2.4.tar.gz-c/home/admin/project/elk
cd/home/admin/project/elkelasticsearch-6.2.4
Vim Config/elasticsearch.yml
Cluster.name:elasticsearch
Node.name:node-1
network.host:10.2.151.203
http.port:9200
Bootstrap.memory_lock:false
Bootstrap.system_call_filter:false
Http.cors.enabled:true
Http.cors.allow-origin: "*"

4.) Start Elasticsearch
Useradd Elk
Chown–r elk.elk/home/admin/project/elk/elasticsearch-6.2.4
./bin/elasticsearch–d

NETSTAT–LUNTP #查看监听端口9200 9300

Curl 10.2.151.203:9200

5.) Start a common error
Uncaught exception in thread [main]
Org.elasticsearch.bootstrap.StartupException:java.lang.RuntimeException:can not run Elasticsearch as root
Problem reason: Cannot start with root user
WORKAROUND: Switch to another user to start

Unable to install Syscall filter:
Java.lang.UnsupportedOperationException:seccomp Unavailable:
Cause: It's just a warning, mainly because your Linux version is too low to cause
WORKAROUND: The warning does not affect use and can be ignored

Error:bootstrap checks failed
Memory locking requested for Elasticsearch process but memory was not locked
Problem reason: Failed to lock memory
WORKAROUND: Switch to the root user and edit the limits.conf configuration file
Vim/etc/security/limits.conf

• Hard Nproc 65536
• Soft Nproc 65536
• Hard Nofile 65536
• Soft Nofile 65536

Max number of threads [1024x768] for user [es] was too low, increase to at least [2048]
Cause: Unable to create local thread problem, user maximum number of threads to create is too small
Solution: Switch to the root user, enter the LIMITS.D directory, modify the 90-nproc.conf configuration file
Vim/etc/security/limits.d/90-nproc.conf

• Soft Nofile 65536
• Soft Nproc 65536
• Soft Nproc 2048

Max virtual Memory Areas Vm.max_map_count [65530] is too low, increase to at least [262144]
Cause: The maximum virtual memory is too small
Solution: Switch to the root user and modify the configuration file sysctl.conf
Vim/etc/sysctl.conf
vm.max_map_count=655360
Sysctl-p

System call filters failed to install; Check the logs and fix your configuration or disable system call filters at your own risk
Cause of the problem: because CENTOS6 does not support Seccomp
Workaround: Configure Bootstrap.system_call_filter to False in Elasticsearch.yml, and note the following in memory:
Bootstrap.memory_lock:false
Bootstrap.system_call_filter:false

2, Elasticsearch-head plug-in installation
View Elasticsearch cluster status information through the Web interface

1.) Download and install Nodejs
wget HTTPS://NODEJS.ORG/DIST/V8.11.3/NODE-V8.11.3-LINUX-X64.TAR.XZ
TAR-ZXVF node-v8.11.3-linux-x64.tar.gz-c/home/admin/project/elk/
cd/home/admin/project/elk/
MV node-v8.11.3-linux-x64/node-v8.11.3
#配置nodejs环境变量
vim/etc/profile.d/node.sh
Export node_home=/home/admin/project/elk/node-v8.11.3
Export path= $NODE _home/bin: $PATH
Export Node_path= $NODE _home/lib/node_modules
source/etc/profile.d/node.sh
#查看nodejs是否生效
[Email protected] node-v8.11.3]$ node-v
v8.11.3
[Email protected] node-v8.11.3]$ npm-v
5.6.0

2.) Install Grunt
NPM Config Set Registry https://registry.npm.taobao.org
Vim ~/.NPMRC
registry=https://registry.npm.taobao.org
Strict-ssl = False
NPM install-g GRUNT-CLI
#将grunt加入系统文件
Ln-s/home/admin/project/elk/node-v8.11.3/lib/node_modules/grunt-cli/bin/grunt/usr/bin/grunt

3.) Download Head binary package
wget Https://codeload.github.com/mobz/elasticsearch-head/zip/master
Unzip Elasticsearch-head-master.zip
CD Elasticsearch-head-master
NPM Install
#如果速度较慢或安装失败, we recommend the use of domestic mirrors
NPM Install--ignore-scripts-g cnpm--registry=https://registry.npm.taobao.org

4.) Modify the Elasticserach configuration file
VI./CONFIG/ELASTICSEARCH.YML
#增加新的参数 so that the head plugin can access ES
Http.cors.enabled:true
Http.cors.allow-origin: "*"

5.) Modify the Gruntfile.js configuration
Vim Gruntfile.js
#port: 9100 add hostname address above
Hostname: "0.0.0.0",

6.) Modify the _site/app.js configuration
Vim _site/app.js
#localhost替换为IP地址
This.base_uri = This.config.base_uri | | This.prefs.get ("App-base_uri") | | "Http://10.2.151.203:9200";

7.) Start Grunt
Grunt Server
#如果启动成功, you can run directly in the background and the command line can continue typing (but if you want to quit, you need to kill the process yourself)
Grunt Server &
Nohup Grunt Server & Exit #后台启动

#启动提示模块未找到

> Local Npm Module "Grunt-contrib-jasmine" not found. Is it installed?
NPM Install Grunt-contrib-jasmine #安装模块

3, Installation Kibana
1.) Download and install
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.2.4-linux-x86_64.tar.gz
TAR-ZXVF kibana-6.2.4-linux-x86_64.tar.gz-c/home/admin/project/elk/
Cd/ho me/admin/project/elk/kibana-6.2.4-linux-x86_64
2.) Modify the configuration
Vim Config/kibana.yml
server.port:5601
Server.host: "IP"
elasticsearch.url:http://ip:9200

3.) Start Kibana
./bin/kibana

4, Installation Logstash
1.) Download and install
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.2.4.tar.gz
TAR-ZXVF logstash-6.2.4.tar.gz-c/home/admin/project/elk/
cd/home/admin/project/elk/logstash-6.2.4
2.) Create a new template
Vim config/test.conf
Input
{
Kafka
{
Bootstrap_servers = "10.7.1.112:9092"
topics = "Nethospital_2"
codec = "JSON"
}
}

Output
{
if [fields][tag] = = "Nethospital_2"
{
Elasticsearch
{
hosts = ["10.7.1.111:9200"]
index = "Nethospital_2-%{+yyyy-mm-dd}"
codec = "JSON"
}
}
}

3.) Start Logstash
Nohup./bin/logstash–f config/test.conf & #-F Specify configuration file
5, Installation Kafka
1.) Download and install
wget https://archive.apache.org/dist/kafka/1.0.0/kafka_2.11-1.0.0.tgz
wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.12/zookeeper-3.4.12.tar.gz
TAR-ZXVF kafka_2.11-1.0.0.tgz-c/home/admin/project/elk/
TAR-ZXVF zookeeper-3.4.12.tar.gz-c/home/admin/project/elk/
cd/home/admin/project/elk/kafka_2.11-1.0.0/

2.) Modify the Kafka parameters and start
Vim Config/zookeeper.properties
Datadir=/tmp/zookeeper/data # Data Persistence path
clientport=2181 # Connection Port
MAXCLIENTCNXNS=100 # Maximum number of connections
Datalogdir=/tmp/zookeeper/logs #日志存放路径
ticktime=2000 # Zookeeper Server heartbeat time, per millisecond
Initlimit=10 # votes to elect the initial time of the new leader.
#启动zookeeper
./bin/zookeeper-server-start.sh Config/zookeeper.properties
#后台启动
Nohup./bin/zookeeper-server-start.sh Config/zookeeper.properties &

3.) Modify the Kafka parameters and start
Vim Config/server.properties
Broker.id=0
port=9092
host.name=10.2.151.203
Num.network.threads=3
Num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
Log.dirs=/data/logs/kafka
num.partitions=2
Num.recovery.threads.per.data.dir=1
log.retention.check.interval.ms=300000
zookeeper.connect=localhost:2181
zookeeper.connection.timeout.ms=6000

#启动zookeeper
./bin/kafka-server-start.sh Config/server.properties
#后台启动
Nohup bin/kafka-server-start.sh Config/server.properties &

4.) Test Kafka
#创建topic (Test)
bin/kafka-topics.sh--create--zookeeper 10.2.151.203:2181--replication-factor 1--partitions 1--topic test

#查看topic
bin/kafka-topics.sh--list--zookeeper 10.2.151.203:2181

#启动生产进程测试
bin/kafka-console-producer.sh--broker-list 10.2.151.203:9092--topic test

#启动启动消费者进程
bin/kafka-console-consumer.sh--zookeeper 10.2.151.203:2181--topic test--from-beginning

6, Installation Filebeat
1.) Download and install
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.2.4-linux-x86_64.tar.gz
TAR–ZXVF Filebeat-6.2.4-linux-x86_64.tar.gz–c/home/admin/project/elk
Cd/home/admin/project/elk/filebeat-6.2.4-linux-x86_64

2.) Configure Filebeat
Vim Filebeat.yml

• Input_type:log
  #Paths that should is crawled and fetched. Glob based paths.
  Paths
• /home/admin/project/other_project/nh-interface/nh-interface.log
  Fields
  Tag:nh-interface
  Multiline
  Pattern: ' ^[0-9]{4}-[0-9]{2}.* '
  Negate:true
  Match:after

Output.kafka:
Enabled:true
Hosts: ["appelk1:9092", "appelk2:9092", "appelk3:9092"]
Topic: ' Hospital '
Compression:gzip
max_message_bytes:100000000

3) Start Filebeat
Nohup./FILEBEAT-E-C Filebeat.yml &

View cluster status
Curl-xget ' Http://10.2.151.203:9200/_cat/nodes '
Curl-xget ' Http://10.2.151.203:9200/_cat/nodes?v '
Curl-xget ' Http://10.2.151.203:9200/_cluster/state/nodes?pretty '

View Cluster Master
Curl-xget ' Http://10.2.151.203:9200/_cluster/state/master_node?pretty '
or Curl-xget ' http://10.2.151.203:9200/_cat/master?v '

Querying the health status of a cluster
Curl-xget ' Http://10.2.151.203:9200/_cluster/health?pretty '
Or
Curl-xget ' Http://10.2.151.203:9200/_cat/health?v '

7. Install Cerebro plug-in
Cerebo is a Kopf on ES5 that manages and monitors elasticsearch cluster state information through a web interface

1.) Download and install
#wget https://github.com/lmenezes/cerebro/releases/download/v0.8.1/cerebro-0.8.1.tgz
#tar –ZXVF Cerebro-0.8.1.tgz/home/admin/project/elk
#cd/home/admin/project/elk/cerebro-0.8.1
# #vim Conf/application.conf
#hosts = [* * * *
{
Host = "http://10.2.151.203:9200"
Name = "My-elk"
},
]

2.) Start/access
Nohup./bin/cerebro & #后台运行
http://10.2.151.203:9000

8. Install Bigdesk plug-in
Bigdesk statistical analysis and charting of Elasticsearch cluster status information
1.) Download and install
#wget Https://codeload.github.com/hlstudio/bigdesk/zip/master
Unzip Bigdesk-master.zip
#mv bigdesk-master/home/admin/project/elk/elasticsearch-6.2.4/plugins/
#cd/home/admin/project/elk/elasticsearch-6.2.4/plugins/bigdesk-master/_site**

2. ) use Python-m simplehttpserver to quickly build HTTP services
Specify Port 8000
Nohup python-m simplehttpserver 8000 & #后台运行

Http://10.2.151.203:8000/

Elk6+filebeat+kafka installation Configuration