Python Elasticsearch API Operation ES Cluster

Source: Internet
Author: User
Tags python elasticsearch

Environment
    • Centos 7.4
    • Python 2.7
    • PIP 2.7 Mysql-python 1.2.5 Elasticsearc 6.3.1
    • Elasitcsearch6.3.2
Knowledge points
    • Calling the Python Elasticsearh API
    • Python MYSQLDB Use
    • DSL Query and Aggregation
    • Pyehon list Operations
Code
#!/usr/bin/env python#-*-coding:utf-8-*-#minyt 2018.9.1# Get the number of modules that occur within 24 hours # The program obtains the relevant refinement data through the Elasticsearch Python client. You can calculate the number of requests, number of timeouts, errors, accuracy, error rate, and so on import mysqldbfrom elasticsearch import elasticsearchfrom elasticsearch import helpers# Define the Elasticsearch cluster index name index_name = "logstash-nginxlog-*" #实例化Elasticsearch类, and set the timeout to 180 seconds, the default is 10 seconds, if the amount of data is large, time is set a bit longer ES = Elasticsearch ([' elasticsearch01 ', ' elasticsearch02 ', ' elasticsearch03 '],timeout=180) #DSL (domain-specific language) query syntax, query Top50 Sname number of permutations Data_sname = {"Aggs": {"2": {"Terms": {"field": "Apistatus.sname.keyword", "Size": 10 0, "order": {"_count": "Desc"}}}, "Size": 0, "_source": {"excludes": []}, "      Stored_fields ": [" * "]," Script_fields ": {}," Docvalue_fields ": [" @timestamp "]," query ": {" bool ": {              "Must": [{"Match_all": {}}, {"range": {"@timestamp": {          "GTE": "now-24h/h", "LT": "now/h"  }}], "filter": [], "should": [], "must_not": []}} #按照DSL (domain-specific language) syntax query get Data de            F get_original_data (): try: #根据上面条件搜索数据 res = Es.search (Index=index_name, Size=0, Body=data_sname) return res except:print "Get Original data failure" #初始化数据库def init_m Ysql (): # Open database Connection db = MySQLdb.connect ("localhost", "myuser", "MyPassword", "MyDB", charset= ' UTF8 ') # Use cursor () Method gets the cursor cursor = db.cursor () # SQL UPDATE statement sql = "Update appname set COUNT=0" Try: # Execute SQL statement curs Or.execute (SQL) # Commit to Database Execution Db.commit () except: # Rollback when an error occurs db.rollback () # Close the database connection DB.C Lose () def updata_mysql (sname_count,sname_list): # Open database Connection db = MySQLdb.connect ("localhost", "myuser", "MyPassword "," MyDB ", charset= ' UTF8 ') # Use the cursor () method to get the cursor cursor = db.cursor () # SQL UPDATE statement sql =" Update appname set CO unt=%d where sname = '%s'% (sname_count,sname_list) Try: # Execute SQL statement cursor.execute (SQL) # Commit to Database execution Db.commit () Except: # rollback Db.rollback () # When an error occurs () # Close the database connection Db.close () #根据Index数据结构通过Elasticsearch Python client uploads the data to the new in Dexdef import_process_data (): try: #列表形式显示结果 res = get_original_data () #print res res_list = Res.get (' aggregations '). Get (' 2 '). Get (' buckets ') #print res_list #初始化数据库 init_mysql () #获取24小时 Sname for value in res_list:sname_list = Value.get (' key ') Sname_count = Value.get (' Doc_ Count ') Print Sname_list,sname_count #更新sname_status值 updata_mysql (sname_count,sname_list ) except Exception, E:print repr (e) if __name__ = = "__main__": Import_process_data ()
Summarize

The key is that the writing of the DSL syntax involves querying and aggregating through Kibana's visualize or Devtool first testing out the correct syntax, and then combining Python with lists, dictionaries, divisions, strings, and so on. The following summarizes each algorithm:

    • Total requests
      Http_host.keyword:api.mydomain.com

    • Extra Long Request
      Http_host.keyword:api.mydomain.com and Request_time: [1 to +] not apistatus.status.keyword:* error

    • Error request
      apistatus.status.keyword:* Error and (http_host.keyword:api.mydomain.com OR http_host.keyword:api.yourdomain.com)

    • Health of Request
      Domain name and request_time aggregation, the domain name request time is less than 3 seconds divided by the total number of requests corresponding to the health of each domain name

    • Request correct rate
      Domain name and HTTP status code aggregation, the domain name HTTP status code is 200 divided by the total number of requests for the domain name of each domain name request correct rate

Python Elasticsearch API Operation ES Cluster

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.