ELK---Log analysis system

Elk is a complete set of log analysis systems

Elk=logstash+elasticsearch+kibana

Unified Official Website Https://www.elastic.co/products

Elk Module Description

Logstash

Role: For processing incoming logs, collecting, filtering, and writing logs

Logstash is divided into three components Input,filter,output

Enter input

Common File,redis,kafka

Example:

Input

File {

Path = ['/var/log/neutron/dhcp-agent.log ']//log path

tags = [' OpenStack ', ' oslofmt ', ' neutron ', ' neutron-dhcp-agent ']

Start_position = "Beginning"

Type = "Neutron"

Codec = Multiline {//merge rows

Pattern = "^%{openstack_trace_block}"//Custom variable

what = "Previous"//previous line merge, next line merge

}

}

Input {

Kafka {

Zk_connect = "server:2181"

topic_id = "Nova"

Codec =>json

Reset_beginning = False

Consumer_threads = 2

Decorate_events =>true

}

}

Filtering filter

Common date time processing, Grok regular capture, GeoIP address query

Example:

fileter{

Grok {

Match + = {"Message" = "%{openstack_normal}%{greedydata:message}"}

overwrite = ["message"]//Rewrite message

}

}

}

Grok built-in variables

Variable can be customized

1. Custom variable Path

/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns

2. Custom variables stored in/opt/logstash/patterns

In the configuration, add

Filter { Grok { Patterns_dir = ["/opt/logstash/patterns"] Match + = {"Message" = "%{openstack_normal}%{greedydata:message}"}} } }

Openstack_normal ^%{timestamp_iso8601:logdate}%{number:pid}%{loglevel:loglevel}%{NOTSPACE:module}%{SPACE}

2016-04-27 15:19:14.455 4392 DEBUG Nova.api.openstack.wsgi [req-fde66cf0-6d28-4b0d-8671-bce33bb48665 0F288A5B5F19437DB670EF94269BFD36 629fb63dd82e46fa937accc99d417059---] Action: ' Create ', calling method: <bound Methodserverscontroller.createof<nova.api.openstack.compute.servers.serverscontroller Object at 0x7c61d10> , body: {"Server": {"name": "Test11", "Imageref": "c9620d95-fc3a-4090-b9e8-6c3909cc556e", "Flavorref": "100000000" , "Max_count": 1, "Min_count": 1, "networks": [{"UUID": "E18F583F-C8CF-433A-8095-315712525ECD"}]}} _process_stack/usr/ lib/python2.7/site-packages/nova/api/openstack/wsgi.py:789

Output

Common Elasticserch, Save as file, output to HDFs, standard output

Example:

Output {

Elasticsearch {

hosts=>["server:9200"]//old version of host new version hosts

Document_type = "%{type}"

Workers = 2

index = "logstash-%{type}-%{+yyyy. MM.DD} "//Index Name

}

}

Output {

Kafka {

Bootstrap_servers = "server:9092"

topic_id = "Nova"

Compression_type = "Snappy"

}

}

Elasticsearch

Used to establish dynamic inverted index of import data, establish disk cache, provide disk synchronization control to achieve quasi-real-time retrieval

DB and Elasticsearch Contrast

Index indexes

An index is equivalent to a library of a database

Type

Type is equivalent to a table in the database

Document

The document corresponds to a row of data in the database

Filed

property is equivalent to a field in a database

Mapping

Mapping is understood as a scenario

Query method

1. query-string

Curl-xget server:9200/logstash-nova-2016.04.27/nova/_search?q=pid.raw:1524 '

2.DSL (Common)

Curl-xget server:9200/logstash-nova-2016.04.27/nova/_search-d ' {

"Query": {

"term": {"Pid.raw": "1524"}

}

}

GET Query

POST Update

PUT creation

Delete Deletes

Head for basic information

Cluster (Cluster)

An ES cluster is a collection of one or more nodes that collectively store the entire data set and provide a federated index and the ability to search across all nodes.

ES cluster needs to modify configuration file

Config/elasticsearch.yml

Cluster.name the same in each ES machine configuration file, Node.name inconsistent

Intra es cluster for HA to avoid single point of failure

The cluster automatically selects a master node, monitors node state, and distributes it to other nodes if a failed Extract node replica shard occurs.

Nodes (node)

The ES host, which runs a single instance, is called a node, which is a member of the cluster that can store data, participate in cluster indexing, and search operations.

Shard (Shard)

A shard store index, an index may exist on multiple shards.

There are two types of shard: Primary and Replica, the Master shard and the replica shard.

Primary Shard is created, the number of its Primary shard will not be changed, the default is 5

Replica Shard is a copy of Primary Shard, which is used for redundant data and improves search performance by default of 1.

Description

Elasticsearch optimization Scheme

1. Configuring data storage locations using SSDs in ELASTICSEARCH.YML

Path.data:/mnt/data/elasticsearch #数据存在挂载硬盘 Configuration

1. Adjust the number of shards according to the number of indexes
2. Adjust the memory according to the actual situation
3. Elasticsearch mapping adjustment, some default settings can be disabled

_all fields are default in index, which facilitates queries, but increases index time and index size

"_all": {"Enabled": false}

EXECUTE statement

PUT My_index

{

"Mappings": {

"My_type": {

"_all": {

"Enabled": false

}

}

}

}

Elasticsearch API Description

Document API: Provides additions and deletions to documents

Search API: Provides a query for a field on a document

Index API: Provides operations on an index

View API: Return data in more intuitive form, more suitable for console request presentation

Cluster API: An API for viewing and manipulating clusters

Query syntax can be referred to the official website for Learning:

Https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html

Install Elasticsearch Head Plugin management es

Kibana

Used to get Elasticsearch data for presentation

Application

Create a log collection for a virtual machine

Logstash configuration file

Input { File { Path = ['/var/log/nova/nova-api.log '] tags = [' OpenStack ', ' oslofmt ', ' oslofmt ', ' Nova ', ' Nova-api '] Start_position = "Beginning" Type = "Nova" } File { Path = ['/var/log/nova/nova-conductor.log '] tags = [' OpenStack ', ' oslofmt ', ' Nova ', ' Nova-conductor '] Start_position = "Beginning" Type = "Nova" } File { Path = ['/var/log/nova/nova-scheduler.log '] tags = [' OpenStack ', ' oslofmt ', ' Nova ', ' Nova-scheduler '] Start_position = "Beginning" Type = "Nova" } File { Path = ['/var/log/nova/nova-compute.log '] tags = [' OpenStack ', ' oslofmt ', ' Nova ', ' Nova-compute '] Start_position = "Beginning" Type = "Nova" } File { Path = ['/var/log/neutron/server.log '] tags = [' OpenStack ', ' oslofmt ', ' neutron ', ' neutron-server '] Start_position = "Beginning" Type = "Neutron" } } Filter { Mutate { gsub = [' path ', '/.+/', ' "] } If "oslofmt" in [tags] { Grok { Match + = {"Message" = "%{openstack_normal}%{greedydata:message}"} overwrite = ["Message"] } } If ' Traceback ' in [message] or ' File ' in [message] or ' runtimeerror ' in [message] or ' Command ' in [message] or ' Exit ' in [ Message] or "Stdin" in [message]{ Multiline { Pattern = "^%{greedydata}" what = "Previous" } } Date { Match = ["Logdate", "Yyyy-mm-dd HH:mm:ss. SSS ", "EEE MMM DD HH:mm:ss. SSSSSS yyyy ", "Dd/mmm/yyyy:hh:mm:ss", "Dd-mmm-yyyy::hh:mm:ss", "MMM dd HH:mm:ss", "MMM dd HH:mm:ss", "Yyyy-mm-dd HH:mm:ss. SSS "] } If [loglevel] in ["WARNING", "WARN", "TRACE", "ERROR"] { Mutate { Add_tag = ["Something_wrong"] } } } Output { stdout {codec = Rubydebug} Elasticsearch {hosts=>["server:9200"] Document_type = "%{type}" Workers = 16 index = "logstash-%{type}-%{+yyyy. MM.DD} " } }

Elasticsearch

Follow the module query DSL statement

Nova-api

Curl-xget "Http://192.168.44.128:9200/logstash-nova-2016.04.27/nova/_search?pretty=true"-D ' {' query ': {"BOOL": {"Must": [{"term": {"Path.raw": "Nova-api.log"}},{"query_string": {"Default_field": "_all", "Query": "Fde66cf0 783b26ba"}}, {"Range": {"Logdate.raw": {"GT": "2016-04-27 15:19:14.455", "LT": "2016-04-27 15:19:21.999"} }}] } } }‘

Nova-scheduler

Curl-xget "Http://192.168.44.128:9200/logstash-nova-2016.04.27/nova/_search?pretty=true"-D ' {' query ': {"BOOL": {"Must": [{"Term": {"Path.raw": "Nova-scheduler.log"}},{"query_string": {"Default_field": "_all", "Query": "Fde66cf0"}}], "Must_not": [], "should": [] } } }‘

Nova-conductor

Curl-xget "Http://192.168.44.128:9200/logstash-nova-2016.04.27/nova/_search?pretty=true"-D ' {' query ': {"BOOL": {"Must": [{"Term": {"Path.raw": "Nova.conductor.log"}}, {"Query_string": {"Default_field": "_all", "Query": "Fde66cf0"}}], "Must_not": [], "should": [] } } }‘

Nova-compute

Curl-xget "Http://192.168.44.128:9200/logstash-nova-2016.04.27/nova/_search?pretty=true"-D ' {' query ': {"BOOL": {"Must": [{"Term": {"Path.raw": "Nova-compute.log"}}, {"Query_string": {"Default_field": "_all", "Query": "Fde66cf0"}}, {"Range": {"Logdate.raw": {"GT": "2016-04-2715:19:10.000", "LT": "2016-04-2715:25:07.981"}}], "Must_not": [{"term": {"Module.raw": "Oslo_service.periodic_task"}}, {"term": {"Module.raw": "Oslo_concurrency.lockutils"}}, {"term": {"Module.raw": "Keystoneclient.session"}}], "should": [] } } }‘

Server.log

Curl-xget "Http://192.168.44.128:9200/logstash-neutron-2016.04.27/neutron/_search?pretty=true"-D ' {' query ': {"BOOL": {"Must": [{"term": {"Path.raw": "Server.log"}}, {"Query_string": {"Default_field": "_all", "Query": "783b26ba Ed1db9be"}}, {"Range": {"Logdate.raw": {"GT": "2016-04-2715:19:10.000", "LT": "2016-04-27 15:21:00.000"} } }] } } }‘

ELK---Log analysis system