Python Elasticsearch API Operation ES Cluster

Environment

• Centos 7.4
• Python 2.7
• PIP 2.7 Mysql-python 1.2.5 Elasticsearc 6.3.1
• Elasitcsearch6.3.2

Knowledge points

• Calling the Python Elasticsearh API
• Python MYSQLDB Use
• DSL Query and Aggregation
• Pyehon list Operations

Code

#!/usr/bin/env python#-*-coding:utf-8-*-#minyt 2018.9.1# Get the number of modules that occur within 24 hours # The program obtains the relevant refinement data through the Elasticsearch Python client. You can calculate the number of requests, number of timeouts, errors, accuracy, error rate, and so on import mysqldbfrom elasticsearch import elasticsearchfrom elasticsearch import helpers# Define the Elasticsearch cluster index name index_name = "logstash-nginxlog-*" #实例化Elasticsearch类, and set the timeout to 180 seconds, the default is 10 seconds, if the amount of data is large, time is set a bit longer ES = Elasticsearch ([' elasticsearch01 ', ' elasticsearch02 ', ' elasticsearch03 '],timeout=180) #DSL (domain-specific language) query syntax, query Top50 Sname number of permutations Data_sname = {"Aggs": {"2": {"Terms": {"field": "Apistatus.sname.keyword", "Size": 10 0, "order": {"_count": "Desc"}}}, "Size": 0, "_source": {"excludes": []}, "      Stored_fields ": [" * "]," Script_fields ": {}," Docvalue_fields ": [" @timestamp "]," query ": {" bool ": {              "Must": [{"Match_all": {}}, {"range": {"@timestamp": {          "GTE": "now-24h/h", "LT": "now/h"  }}], "filter": [], "should": [], "must_not": []}} #按照DSL (domain-specific language) syntax query get Data de            F get_original_data (): try: #根据上面条件搜索数据 res = Es.search (Index=index_name, Size=0, Body=data_sname) return res except:print "Get Original data failure" #初始化数据库def init_m Ysql (): # Open database Connection db = MySQLdb.connect ("localhost", "myuser", "MyPassword", "MyDB", charset= ' UTF8 ') # Use cursor () Method gets the cursor cursor = db.cursor () # SQL UPDATE statement sql = "Update appname set COUNT=0" Try: # Execute SQL statement curs Or.execute (SQL) # Commit to Database Execution Db.commit () except: # Rollback when an error occurs db.rollback () # Close the database connection DB.C Lose () def updata_mysql (sname_count,sname_list): # Open database Connection db = MySQLdb.connect ("localhost", "myuser", "MyPassword "," MyDB ", charset= ' UTF8 ') # Use the cursor () method to get the cursor cursor = db.cursor () # SQL UPDATE statement sql =" Update appname set CO unt=%d where sname = '%s'% (sname_count,sname_list) Try: # Execute SQL statement cursor.execute (SQL) # Commit to Database execution Db.commit () Except: # rollback Db.rollback () # When an error occurs () # Close the database connection Db.close () #根据Index数据结构通过Elasticsearch Python client uploads the data to the new in Dexdef import_process_data (): try: #列表形式显示结果 res = get_original_data () #print res res_list = Res.get (' aggregations '). Get (' 2 '). Get (' buckets ') #print res_list #初始化数据库 init_mysql () #获取24小时 Sname for value in res_list:sname_list = Value.get (' key ') Sname_count = Value.get (' Doc_ Count ') Print Sname_list,sname_count #更新sname_status值 updata_mysql (sname_count,sname_list ) except Exception, E:print repr (e) if __name__ = = "__main__": Import_process_data ()

Summarize

The key is that the writing of the DSL syntax involves querying and aggregating through Kibana's visualize or Devtool first testing out the correct syntax, and then combining Python with lists, dictionaries, divisions, strings, and so on. The following summarizes each algorithm:

• Total requests
  Http_host.keyword:api.mydomain.com

• Extra Long Request
  Http_host.keyword:api.mydomain.com and Request_time: [1 to +] not apistatus.status.keyword:* error

• Error request
  apistatus.status.keyword:* Error and (http_host.keyword:api.mydomain.com OR http_host.keyword:api.yourdomain.com)

• Health of Request
  Domain name and request_time aggregation, the domain name request time is less than 3 seconds divided by the total number of requests corresponding to the health of each domain name

• Request correct rate
  Domain name and HTTP status code aggregation, the domain name HTTP status code is 200 divided by the total number of requests for the domain name of each domain name request correct rate

Python Elasticsearch API Operation ES Cluster