Objective
process, NIGNX format log into JSON, Logstash directly to Elasticsearch, and then through the Kibana GUI interface display analysis
Important NIGNX Log into JSON format, avoid nignx default log is a space, need a regular match, resulting in logstash too much CPUThe Elasticsearch machine configures the firewall, allowing only the specified Logstash machine accessKibana only listens for local 127.0.0.1 use NIGNX direction Agent, Nginx Config
Recently in the log analysis of this piece, to use Logstash+elasticsearch+kibana to implement log import, filtering and visual management, official documentation is not detailed enough, the online articles are mostly either for the use of Linux systems, or the copying of other people's configuration is mostly impossible to run. It took a lot of effort to get rid of these three things, write a usage experience, nonsense not much to say, into the subjec
First heard elk, is Sina's @argv introduction internal use elk situation and scene, at that time touched very big, originally have so convenient way to collect log and show, have such tool, you do bad thing, delete log, it has no effect.A lot of companies say they are concerned about security, but they have not seen and watched the logs of their servers, which is a bit ironic. Manage the logs first, and then we'll discuss security in depth.Mirantis's fuel, has introduced elk as a monitoring tool
1. Workflow of Log Platform650) this.width=650; "src=" http://s3.51cto.com/wyfs02/M01/71/5F/wKioL1XNWHGwPB_ZAAErAE7qZjQ757.jpg "title=" 1.png " alt= "Wkiol1xnwhgwpb_zaaerae7qzjq757.jpg"/>
shipper means log collection, using Logstash to collect log data from various sources, such as system logs, files, Redis, MQ, and so on;
broker as a buffer between the remote agent and the central agent, using Redis implementation, one can improve the performance of the system, the secon
Installation and deployment one, environment configuration
Operating system: Cent OS 7
Kibana version: 3.1.2
JDK version: 1.7.0_51
SSH Secure Shell version: Xshell 5
Second, operation process 1: Download the specified version of the KibanaGo to the installation directory and download the Kibana compressed package file and unzip it via the Curl command:
Download
Curl-l
Overview
Log System Elk use details (i)-How to useLog System Elk use details (ii) –logstash installation and useElk Use of log system (iii) –elasticsearch installationLog System Elk use details (iv) –kibana installation and useElk Use of log system (v)-supplement
This is the last of the small series, and we'll see how to install Kibana and make a quick query about the log information in elk.
Fluentd is an open source collection event and log system that currently offers 150 + extensions that let you store big data for log searches, data analysis and storage.
Official address http://fluentd.org/plugin address http://fluentd.org/plugin/
Kibana is a Web UI tool that provides log analysis for ElasticSearch, and it can be used to efficiently search, visualize, analyze, and perform various operations on logs. Official Address http://www.elastic
Elasticsearch and Kibanna's link address: https://www.elastic.co/downloads, The packages I use in my environment are kibana-4.1.1-linux-x64.tar.gz and elasticsearch-1.7.1.zip.Installing ElasticsearchIt is assumed that the Java environment has been configured before, so it can be installed directly.[Email protected] ~]# Unzip Elasticsearch-1.7.1.zip[Email protected] ~]# MV Elasticsearch-1.7.1/usr/local/elasticsearchInstalling
adding or modifying inputs, outputs, and filters in your profile, thus making it easier to tailor a more reasonable storage format for the query.Integrated Elasticsearch Insert data above steps have been successfully built Logstash, then add logstash configuration file, so that its configuration file start, the data into ES, display1. Add logs.conf under the/root/config/directoryinput{file{type = "all" Path = "/root/tomcat7/logs/catalina.out"} file{type =gt ; "Access" path = "/root/tomcat7/logs
Kibana + Logstash + Elasticsearch log query system, kibanalostash
The purpose of this platform is to facilitate log query During O M and R D. Kibana is a free web shell. Logstash integrates various log collection plug-ins and is also an excellent regular-cut log tool. Elasticsearch is an open-source search engine framework (supporting cluster architecture ).
1 installation requirement 1.1 theoretical Topo
Document directory
4. Performance Tuning
The purpose of this platform is to facilitate log query During O M and R D. Kibana is a free web shell. logstash integrates various log collection plug-ins and is also an excellent regular-cut log tool. elasticsearch is an open-source search engine framework (supporting cluster architecture ).
1 installation requirement 1.1 theoretical Topology
1.2 installation environment 1.2.1 hardware environment
192
The purpose of building this platform is to facilitate the operation of the research and development of the log query. Kibana a free web shell; Logstash integrates various collection log plug-ins, or is a good regular cutting log tool; Elasticsearch an open-source search engine framework that supports the cluster architecture approach.1 Installation Requirements 1.1 theoretical topology1.2 Installation Environment 1.2.1 hardware environment192.168.50.
Do Android 3 years, the network is not very concerned about, and now look let me eat a surprise, many of the previously expected features are open source, and powerful, try a bit. Simple trialDownload elasticsearch-1.4.2 and startDownload logstash-1.4.2, run the following commandBin/logstash-e ' input {stdin {}} output {elasticsearch {host = localhost}} 'The data entered by the console will be logstash to Elasticsearch (so Logstash is a data glue or connector)Found a more index (but also a usele
Kibana do not select the field you want to select, that is, the term to filter the selected field when the Discovery list does not have this option.650) this.width=650; "src=" Http://s3.51cto.com/wyfs02/M01/79/08/wKiom1aFAWuSYiPXAAAaSCMrdEo742.gif "style=" float: none; "title=" 3.gif "alt=" Wkiom1afawusyipxaaaascmrdeo742.gif "/>Go to discover to see, found that this field is preceded by a question mark, click to prompt this field is not indexed, not f
=" Head.png "alt=" Wkiom1esmxwaogtzaajhix4lznm047.png "/>Marvel Plugin : The first step on es: bin/plugin install license, bin/plugin install elasticsearch/marvel/latest (all es are installed) /c4>The second section is in the bin directory of the Kibana: Kibana plugin--install elasticsearch/marvel/latest650) this.width=650; "src=" Http://s5.51cto.com/wyfs02/M01/85/AE/wKioL1esMyizuCSsAAK1nD-zT9g214.png "titl
First, the visualize function of KibanaThe Visualize tab on the home page is used to design visual graphics. You can save the previous search in discovery to make a drawing, then save the visualize, or load the merge into dashboard. A visualization can be based on the following types of data sources:A new interactive searchA saved searchA saved visualizationHere are some of the types of visualize that Kibana comes withType useArea chart uses block dia
Filebeat is a lightweight, open source shipper for log file data. As the next-generation Logstash forwarder, filebeat tails logs and quickly sends this information to Logstash fo R further parsing and enrichment or to Elasticsearch for centralized storage and analysis.
Filebeat than Logstash seems better, is the next generation of log collectors, ELK (Elastic +logstash + Kibana) later estimated to be renamed EFK.
Filebeat How to use:
1, download the
you to collect, analyze, and store your logs for later use (e.g., search).
Kibana is also an open source and free tool, and he kibana a friendly Web interface for Logstash and Elasticsearch, which can help you summarize, analyze, and search important data logs.
Elk work flow is as follows:
Deploy Logstash on all services that need to collect logs, as Logstash agent (Logstash shipper) to monitor and filter
Nlog, Elasticsearch, Kibana and LogstashObjectiveRecently in the document management, it is necessary to record each administrator and user in the process of all operational records, originally through the EF directly to record the operation data in the database, when the query directly from the database read, but this is too stupid, so found on the Internet Logstash this NB tool, Share with you the process of learning.Environment preparationThese thr
The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion;
products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the
content of the page makes you feel confusing, please write us an email, we will handle the problem
within 5 days after receiving your email.
If you find any instances of plagiarism from the community, please send an email to:
info-contact@alibabacloud.com
and provide relevant evidence. A staff member will contact you within 5 working days.