Elasticsearch,kibana,logstash,nlog Implementing ASP. NET Core Distributed log System
Elasticsearch official website
Elasticsearch Documentation
NLog.Targets.ElasticSearch Package
Elasticsearch-Introduction
Elasticsearch, as a core part, is a document repository with powerful indexing capabilities and can be used to search for data through the REST API.
It is written in Java, based on Apache Lucene, although these details are hidden in the API.
By indexed fields, any document that is stored (indexed) can be found in many different aggregations.
However, Elasticsearch not only provides powerful search capabilities for these indexed documents.
Fast, distributed, horizontally scalable, supports real-time document storage and analysis, and supports hundreds of of servers and petabytes of indexed data.
At the same time as the core of Elastic stack (aka ELK), it offers powerful applications such as LogStash, Kibana and more.
Kibana is a Web application dedicated to providing powerful visual queries in Elasticsearch.
With Kibana, you can easily create queries, charts, and dashboards for the data indexed in Elasticsearch.
Elasticsearch opens up a REST API and you'll find many document examples are HTTP calls that you can try using tools such as curl or postman.
Of course, the client for this API has been written in many different languages, including. Net, Java, Python, Ruby, and JavaScript.
Logstash is an open source data collection engine with real-time processing power. Data can be collected dynamically from different sources, and processed (filtered, distorted) through a unified output to a specific address to prepare for more diversified data analysis in the future.
Installing the Java Environment installation
Lazy One-click installation
Yum Install java*
View JDK Version information
Java-version
Elasticsearch Installation
The official website has many installs the way, I here uses the RPM installment, everybody can install according to oneself Custom way,.
Go to local directory cd/usr/local//create Elasticsearch folder mkdir elasticsearch//go to elasticsearch folder cd elasticsearch//start download wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.5.0.rpm//Start Installation Rpm-ivh elasticsearch-5.5.0.rpm
Configuration
Find the installation directory Whereis elasticsearch//Enter the installation directory cd/etc/elasticsearch//edit the Profile VI elasticsearch.yml
Main configuration network.host (native IP) and http.port (default 9200) (current single-node mode, other parameters refer to official documentation)
Start the service
Open port 9200firewall-cmd--add-port=9200/tcp--permanent//Reload configuration firewall-cmd--reload//Set service boot up systemctl enable elasticsearch//start service Systemctl start Elasticsearch
Open http://192.168.30.128:9200 in the browser, as shown to indicate that the boot was successful
Kibana
Kibana Documentation
Installation
Official, official Installation Tutorials
Go to elasticsearch directory cd/usr/local/elasticsearch//download Kibana rpm 64-bit package wget https://artifacts.elastic.co/downloads/ kibana/kibana-5.5.0-x86_64.rpm//Installing KIBANARMP-IVH kibana-5.5.0-x86_64.rpm
Configuration
Go to the installation directory cd/etc/kibana//edit the configuration file VI kibana.yml
Set port number: 5601,host Address: "192.168.30.128", Elasticsearch service Address: "HTTP://192.168.30.128:9200"
Start the service
Open port 5601firewall-cmd--add-port=5601/tcp--permanent//Reload configuration firewall-cmd--reload//Set service boot up systemctl enable kibana//start service Systemctl start Kibana
Open http://192.168.30.128:5601 in Browser, will go to Kibana management interface
LogStash
Logstash Documentation
Installation
Official Official Installation Tutorials
Go to elasticsearch directory cd/usr/local/elasticsearch//download logstash rpm package wget https://artifacts.elastic.co/downloads/ logstash/logstash-5.5.0.rpm//Installing RPM Package RPM-IVH logstash-5.5.0.rpm
Configuration
Enter the installation directory cd/etc/logstash//enter the CONF.D directory CD conf.d//new configuration Information VI nlog.conf
Input: Using TCP to monitor native 8001 port messages
Filter: Use Grok plugin, custom message format, recommended to use Grokdebug online debugging
Output: Using Elasticsearch as the data store
Note: There are very rich plugins for message processing, which can be viewed in official documents.
Start the service
Open port 8001firewall-cmd--add-port=8001/tcp--permanent//Reload configuration firewall-cmd--reload//Set boot up systemctl enable logstash/ /Start Logstashsystemctl start Logstash
Asp. ENT core combined with Nlog for logging
The following is the focus of this article, through Nlog logging, sending messages to Logstash,logstash to store the converted messages to Elasticsearch, and for use in Kibana queries.
Creating an ASP. NET Core Project
This article was created by VS2017. Netcore 1.1 Project Elasticsearch.quickstart
Installing Nlog dependency Packages through NuGet
NLog.Web.AspNetCore
Nlog.Extensions.Logging (pre version)
Add Nlog service in Startup.cs
New Nlog configuration (Web root directory)
<?xml version= "1.0" encoding= "Utf-8"? ><nlog xmlns= "Http://www.nlog-project.org/schemas/NLog.xsd" xmlns: Xsi= "Http://www.w3.org/2001/XMLSchema-instance" autoreload= "true" internalloglevel= "Warn" internallogfile= " Internal-nlog.txt "> <extensions> <!--enable nlog.web for ASP core--> <add assembly=" Nlog.web . Aspnetcore "/> </extensions> <!--define various log targets-<!--definition log file directory--<variable nam E= "Logdirectory" value= "${basedir}/logs/${shortdate}"/> <variable name= "nodeName" value= "Node1"/> < Targets async= "true" > <!--all logs target---<target xsi:type= "File" name= "Allfile" f Ilename= "${logdirectory}/nlog-all/${shortdate}.log" layout= "#node1 #${longdate}#${logger}#${uppercase:${level}}# ${callsite}#${callsite-linenumber}#${aspnet-request-url}#${aspnet-request-method}#${aspnet-mvc-controller}#${ Aspnet-mvc-action}#${message}#${exception:format=tostring}# "keepfileopen=" false "/> <!--local file log target---<target xsi:type=" file "Name=" Ownlog-file "filename=" ${logdirectory}/nlog-${level}/${shortdate}.log "layout=" #$ {Longdate}#${nodename}#${logger}#${uppercase:${level}}#${callsite}#${callsite-linenumber}#${aspnet-request-url }#${aspnet-request-method}#${aspnet-mvc-controller}#${aspnet-mvc-action}#${message}#${exception:format= tostring}# "keepfileopen=" false "/> <!--TCP Log target---<target xsi:type=" networ K "Name=" Ownlog-tcp "keepconnection=" false "address =" tcp://192.168.30.128:8001 " Layout= "#${longdate}#${nodename}#${logger}#${uppercase:${level}}#${callsite}#${callsite-linenumber}#${ aspnet-request-url}#${aspnet-request-method}#${aspnet-mvc-controller}#${aspnet-mvc-action}#${message}#${ exception:format=tostring}# "/> <!--grok Rules--<!--%#{data:request_time}#%{data:node_name}#%{data:class_name}#%{data:log_level}#%{data:call_site}#%{data:line_number}#%{ Data:request_url}#%{data:request_method}#%{data:container_name}#%{data:action_name}#%{data:log_info}#%{data: exception_msg}#--> <!--Blank--<target xsi:type= "Null" name= "blackhole"/> </targets> <!--day Log level Trace-"debug-" Info-"warn-" error-"Fatal--> <!--log Rules--<rules> <!--all logs, including Microsoft log--> ; <logger name= "*" minlevel= "Trace" writeto= "Allfile"/> <!--custom log, excluding Microsoft Logs--<logger name= "MICR Osoft.* "minlevel=" Trace "writeto=" Blackhole "final=" true "/> <logger name=" * "minlevel=" Debug "writeto=" ownLog- File "/> <logger name=" * "minlevel=" Info "writeto=" ownlog-tcp "/> </rules></nlog>
Note: The address in TCP target points to the addresses that are listening in Logstash, and the Grok template information is also given in the comments.
Test Nlog Log records
See the final effect in Kibana
Reference
1:logstash+elasticsearch Simple to use (CentOS)
2: Visualize data using elasticsearch,kibana,asp.net core and Docker
3:elastic Stack and Product documentation
4:elasticsearch installation and configuration on CentOS 7
5:nlog Official documents
6: Build a ELKB Log collection system from scratch
Elasticsearch,kibana,logstash,nlog Implementing ASP. NET Core Distributed log System