First, the use of the backgroundWhen the production environment has many servers, many business module logs need to be viewed every momentSecond, the environmentSystem: CentOS 6.5jdk:1.8Elasticsearch-5.0.0Logstash-5.0.0kibana-5.0.0Third, installation1. Installing the JDKDownload jdk:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.htmlThis environment downloads a 64-bit tar.gz package that copies the installation package to the installation Server/usr/local director
extend the key,value of the A=bc=d in the request, and use the non-schema feature of ES to ensure that if you add a parameter, it can take effect immediately.
UrlDecode is to ensure that the parameters have Chinese words to UrlDecode
Date is the time of day for the document to be saved in ES, otherwise the time to insert ES
Well, now that the structure is complete, you can access the log of this access at the Kibana console once
) unlimitedOpen files (-N) 65535Pipe Size (bytes,-p) 8POSIX message queues (bytes,-Q) 819200Real-time priority (-R) 0Stack size (Kbytes,-s) 10240CPU time (seconds,-t) unlimitedMAX User Processes (-u) 127447Virtual Memory (Kbytes,-V) UnlimitedFile locks (-X) Unlimited# vi/etc/security/limits.d/90-nproc.conf* Soft Nproc 320000Root Soft Nproc UnlimitedThird, ES status is yellowES is represented in three color states: Green,yellow,red.Green: All primary shards and replica shards are availableYellow:
:00.450z ", " host "= " noc.vfast.com "} You can use the Curl command to see if ES has received dataCurl ' Http://localhost:9200/_search?pretty '3, install KibanaUnzip to the corresponding folder after downloading TAR-ZXF kibana-4.1.1-linux-x64.tar.gz-c/usr/local/Start /usr/local/kibana-4.1.1-linux-x64/bin/kibanaWith http://kibanaServerIP:5601 access to Kibana
ELK installation configuration is simple, there are two points to be aware of when managing OpenStack logs:
Logstash configuration file Writing
Capacity planning for Elasticsearch log storage space
Also recommended Elkstack Chinese guide.
ELK IntroductionELK is an excellent open-source software for log collection, storage and querying, and is widely used in log systems. When the Open
appearsConfigure Logstash, CD to the lower bin directory of the Logstash folderCreate the configuration file logstash.conf, as follows:input{ stdin { }}output{ elasticsearch { = =["127.0.0.1:9200" ] index= "Logstash-%{+YYYY. MM.DD} " = + " form "= = "%{id} " } stdout { = json_lines }}Here are the pits:1) Edit file best Choice Notepad open must be UTF-8 Withou BOMThe correct solution is as follows:Installation steps:CD to Logstash folder under Bin dire
0, Preface
This article is mainly referred to dockerinfo this article Elk log system, which Docker configuration file is mainly provided by the blog, I do just on the basis of this article, deleted part of this article does not need, while noting the construction process of some problems.
About Elk, this article does not do too much introduction, detailed can view the official website, here first posted our
:9092,10.82.9.204:9092"
topics => ["filebeat_docker_java"]
}
}
filter {
json {
source => "message"
}
date {
match => ["timestamp","UNIX_MS"]
target => "@timestamp"
}
}
output {
elasticsearch {
hosts => ["10.82.9.205", "10.82.9.206", "10.82.9.207"]
index => "filebeat-docker-java-%{+YYYY.MM.dd}"
}
}
Basic configuration is simple, do not explain too much, through the simple configuration above can be implemented in any
This blog installed Elk version of the current version of the latest 6.3.0, because Elasticsearch is based on Java development, so the JDK version is required, in the 5.0 version, requires JDK version of not less than 1.8 can be normal and practical.At the same time, Elasticsearch,logstash,kibana Three versions are best consistent, otherwise there will be errors due to version conflicts.Start the installati
://ip:9200/_plugin/kopf to view cluster statusInstalling Kibanawget https://download.elastic.co/kibana/kibana/kibana-4.4.0-linux-x64.tar.gzModify the KIBANA.YML configuration (mainly modify the IP of the Elasticsearch)Open ip:5601 to see if the installation was successfulInstalling Logstashwget https://download.elastic.co/logstash/logstash/logstash-2.2.2.tar.gzSi
First, Introduction1. Core compositionELK Consists of three parts: Elasticsearch,Logstash and Kibana ;Elasticsearch is an open source distributed search engine, it features: distributed, 0 configuration, automatic discovery, Index auto-shard, index copy mechanism, RESTful style interface, multi-data source, automatic search load, etc.Logstash is a fully open source tool that collects, analyzes, and stores your logs for later useKibana is an open sou
my Linux version is too low to cause, can be ignored.
CD Elasticsearch-6.0.0-alpha2/bin
./elasticsearch
1.5. Detect if es are running successfully,
Open a new terminal
Curl ' Http://localhost:9200/?pretty '
Note: This means that you have now started and run a Elasticsearch node, and you can experiment with it.A single node can act as an instance of a running elasticsearch. A cluster is a group of nodes with the same cluster.name that can work together and share data, and also provide fault t
Benefits of the unified collection of real-time logs:1. Quickly locate the problem machine in the cluster2, no need to download the entire log file (often relatively large, download time is much)3, the log can be countedA, to find the most frequently occurring anomalies, for tuning processingB, Statistics crawler IPC, Statistical user behavior, do cluster analysis, etc.Based on the above requirements, I adopted the ELK (Elasticsearch + Logstash +
Elasticsearch, Fluentd and Kibana: Open source log search and visualization schemeOffers: Zstack communityObjectiveThe combination of Elasticsearch, Fluentd and Kibana (EFK) enables the collection, indexing, searching, and visualization of log data. The combination is an alternative to commercial software Splunk: Splunk is free at the start, but charges are required if there is more data.This article descri
ELK classic usage-enterprise custom log collection cutting and mysql module, elkmysql
This article is included in the Linux O M Enterprise Architecture Practice Series1. Collect custom logs of cutting companies
The logs of many companies are not the same as the default log format of the service. Therefore, we need to cut the logs.1. sample logs to be cut
11:19:23, 532 [143] DEBUG performanceTrace 1145 http://api.114995.com: 8082/api/Carpool/QueryMatc
This article translates Timrose's article, the original address: https://www.timroes.de/2016/02/21/writing-kibana-plugins-custom-applications/
Before you read this tutorial, you need to read part 1th-the basics.
This tutorial series describes how to create an application in Kibana. An application is a plug-in that is part of the Kibana platform and can place anyt
These three functions are log collection, index and search, and visualized display.
L logstash
This architecture diagram shows that logstash is only the place where collect and index are located. A. conf file is input during runtime, And the configuration is divided into three parts: input, filter, and output.
L redis
Redis serves as a decoupling between log collection and indexing.
L elasticsearch
Core Component used for searching. Main features: Real-Time, distributed, highly available, docum
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.