[Elastalert] Introduction and installation of -1_elasticsearch

It's written in front of you.

Recently started the operation of the maintenance of the work, code modification, build elk, build alarms, here do posting summary. Environment Introduction

Ubuntu14
Elasticsearch 5.1.2
Kibana 5.1.2 Installation

Website website:

Https://elastalert.readthedocs.io/en/latest/running_elastalert.html#tutorial

Perform:

git clone https://github.com/Yelp/elastalert.git 
cd elastalert
python setup.py install            //may require sudo
Pip install-r requirements.txt    //may require sudo
CP config.yaml.example Config.yaml

Note that if you are using elk5.0, Elastalert Master does not support it, you need to switch branches to SUPPORT_ES5 structure introduction

The installation is complete with three commands:

Elastalert-create-index Elastalert will store execution records in an ES index, which is used to create the index, by default, the index is named Elastalert_status. There are 4 _type, each with its own @timestamp field, so you can also use Kibana to view the logging of this index.

Note: In fact, the kibana5.0 does not open this index, it may be a compatibility issue, in the future to find a way.

Elastalert-rule-from-kibana reads filtering settings from the Kibana3 saved dashboard to help generate the configuration in Config.yaml. Note, however, that it will only read filtering, excluding queries.
No use.

Elastalert-test-rule tests the rule settings in the custom configuration.
Note: ES5 does not support the basic use of test functionality

Run the command to load all rules:

Python-m Elastalert.elastalert--config./config.yaml

Or execute a rule in the Rules_folder alone:

Python-m elastalert.elastalert--config./config.yaml--rule./examele_rules/one_rule.yaml

Elastalert supported warning Types Command Email JIRA Opsgenie SNS hipchat Slack Telegram Debug

See: Http://elastalert.readthedocs.io/en/latest/ruletypes.html#alert-content Compounding structure

The

is similar to watcher (or only this way), and the Elastalert configuration structure is divided into several parts, but it has its own name. (Watcher quickly out an official tool, the pure interface is too tired) config.ymal configuration item Rules_folder: The setting to load the next stage rule, the default is Example_rules Run_ Every: Used to set a timed send request to the Elasticsearch Buffer_time: used to set the scope of the Time field in the request, the default is 45 minutes Es_host:elasticsearch host address Es_ Port:elasticsearch corresponding port number use_ssl: Optional, choose whether to use SSL to connect Es,true or false verify_certs: Optional, verify TLS certificate, set to TRUE or FALSE, default to-True Es_username:es Certified username es_password:es certified password es_url_prefix: Optional, Es URL prefix (my understanding is HTTPS or HTTP) es_send_get_body _as: Optional, query es in the way that the default is get Writeback_index:elastalert generated log in Elasticsearch the index created in the
Alert_time_limit: time limit for failed retries

See: Rule rules for Http://elastalert.readthedocs.io/en/latest/ruletypes.html#rule-configuration-cheat-sheet Elastalert Name: Configuration, each rule needs to have its own separate name, and once repeated, the process will not start. Type: Configure, select a data validation method. Index: Configuration, read data from some kind of index, currently support YMD format, need to set use_strftime_index:true first, then match index, configuration like: index:logstash-es-test%y.%m.%d, Indicates the index that matches the beginning of the name of the Logstash-es-test, with the date of the suffix. Filter: Configure to set the filter condition to the ES request. Timeframe: Configuration, Cumulative trigger alarm when the length. Alert: Configure what alert means to execute when triggering an alert. Different type also has its own unique configuration options. At present Elastalert have the following kinds of RuleType:
Any: Alarm as long as there is a match; the contents of the Blacklist:compare_key field match anything in the blacklist array; the contents of the Whitelist:compare_key field could not match the contents of the whitelist array; Change: In the same query_key condition, the contents of the Compare_key field are sent within the timeframe range; frequency: Under the same query_key conditions, timeframe is within the range num_ Events a filtered anomaly; Spike: Under the same query_key condition, the ratio of data in the range of two timeframe is over Spike_height. which can be set through the Spike_type specific direction is-up,down,both. You can also set a lower limit on the amount of data in the previous cycle by threshold_ref setting, which requires the lower bound of the current cycle data quantity, and does not trigger if the amount of data is not lower than the threshold_cur; Flatline:timeframe Range, The amount of data is less than the threshold threshold, and the maximum number of terms_size (default 50) results in the range of terms_window_size (default 30 days) before the New_term:fields field is newly present; cardinality: in the same Under query_key conditions, the Cardinality_field value in the timeframe range exceeds max_cardinality or lower than the Min_cardinality test demo

Background: I have used Docker to build the elk, the log has been connected, on the alarm.
According to the priority, there is a requirement, that is, log status > 500 time on the line alarm.

First on the Kibana to debug the query statement, relatively simple

Status: >=500

Edit the Config.yaml file under Elastalert

 rules_folder:example_rules run_every:seconds:5 #代表每5秒钟轮询query es. # Elastalert'll buffer results from the most recent # Period of ' time ', in case some log sources are

Er_time:minutes:5 #因为日志进入elk会有延迟, you can configure query's forward time range, which is 5 minutes, that is, query time[now-5m, now, which includes a relatively wide number of logs. # The Elasticsearch hostname for metadata writeback # # and every rule can have it ' s own Elasticsearch host Es_host:1 92.168.1.100 es_port:9200 #邮箱告警必配 smtp_host:smtp.sina.com smtp_port:465 #保存了邮箱验证的账号密码信息 smtp_auth_file:example_rules 
/smtp_auth_file.yaml from_addr:xxxx@sina.com use_ssl:false # Option Basic-auth username and password for Elasticsearch #es_username: Someusername #es_password: Somepassword #elastalert need to save data in Es, which is a useful index name and can be default Writeback_index:elastalert 
_status # If A alert fails for some reason, Elastalert'll retry # sending the alert until this time period has Alert_time_limit:days:2 

The contents of the

Edit Example_rules/example_test.yaml file are as follows:

# Alert when the rate of events exceeds a threshold # (Optional) # Elasticsearch host #es_host: 192.168.1.100 # (Optiona L) # Elasticsearch Port #es_port: 9200 # (OptionaL) Connect with SSL to Elasticsearch #use_ssl: false # (OptionaL) basic -auth username and password for elasticsearch #es_username: someusername #es_password: Somepassword # (Required) # rule N
Ame, must be unique Name:name_alert_qycloud_status_error # (Required) # Type of alert.  # The Frequency rule type alerts when num_events events occur with timeframe Time Type:frequency # (Required) # Index to Search, wildcard supported # The index used by this rule query index:monitor-* # (Required, frequency specific) # Alert when this many documen TS matching the query occur within a timeframe Num_events:5 # (Required, frequency specific) # num_events must occur wit Hin this amount of A/trigger an alert Timeframe:minutes:5 # (Required) # A List of elasticsearch filters used F Or Find events # This filters are joined with and NEsted in a filtered query # for more info:http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/ Query-dsl.html # Here is the es5 of the writing filter:-query_string:query: "Status: >=500" # (Required) # The alert is use when a Match is found alert: #-"Email"-"Debug"-"Command" # I'm using the alarm. Use the command line to customize better pipe_match_json:true command: [/home/df/elast Alert/php_alert.php "] # (required, email specific) # A list of email addresses to send alerts to email:-" xxxxx@qq.com "

The rule above shows that in a 5-minute window, if status: >=500 reaches 5 times, the alarm is done.

Start Elastalert service, monitor Elasticsearch

Python-m Elastalert.elastalert--verbose--rule Example_rules/example_test.yaml

Look at the operating situation:

INFO:elastalert:Sleeping for 4 seconds
INFO:elastalert:Queried rule name_alert_qycloud_status_error from 2017-01-29 20:05 CST to 2017-01-29 20:10 cst:6/6 hits INFO:elastalert:Alert for
name_alert_qycloud_status_error at 2017-01-29T12:10:21.651Z:
INFO:elastalert:name_alert_qycloud_status_error at

least 5 events occurred Between 2017-01-29 20:05 CST and 2017-01-29 20:10 CST

@read_timestamp: 2017-01-29t12:10:24.043z
@timestamp: 2017-01-29t12:10:21.651z
@version: 1
_id:avnqgcu6geg-kkwj4pkc
_index:monitor-2017.01.29
_type: Json_php_monitor
app:aysaas-master
beat: {
    "hostname": "Dfdemacbook-air.local", 
    "name": " Dfdemacbook-air.local ", 
    " version ":" 5.1.1 "
}
client:127.0.0.1
elapsed:0
ent_id: 
Error: ...
.......

When you look at the contents of the PHP script used by the alarm, just save the first log message of the alarm:

$fp = fopen (' Php://stdin ', ' R ');
$result = ';

while (!feof ($fp)) {
    $result. = Fgets ($fp, 128);
}
Fclose ($FP);

File_put_contents ('/tmp/alert_test ', $result. "\ r \ n");

About email Alert Compounding

Have not yet been prepared successfully.
Main concerns of the compounding parameters:

#邮箱告警必配
smtp_host:smtp.sina.com
smtp_port:465
#保存了邮箱验证的账号密码信息
smtp_auth_file:example_rules/ Smtp_auth_file.yaml
from_addr:xxxx@sina.com

Smtp_auth_file.yaml content:

User:xxx
password:xxx

Reference Web site

Http://kibana.logstash.es/content/elasticsearch/other/elastalert.html
Http://blog.csdn.net/pujiaolin/article/details/52252950?locationNum=3
http://www.chinacloud.cn/show.aspx?cid=16&id=20922