Python Elasticsearch api, elasticsearchpython
Description: ElasticSearch is a Lucene-based search server. It provides a distributed full-text search engine with multi-user capabilities, based on RESTful web interfaces. The following describes how to use the Python API to query data to facilitate calls by other systems. Install API
pip install elasticsearch
Create an es connection
from elasticsearch import Elasticsearches = Elasticsearch([{'host':'10.10.13.12','port':9200}])
Data Retrieval
es.search(index='logstash-2015.08.20', q='http_status_code:5* AND server_name:"web1"', from_='124119')
Common Parameters
- Index-index name
- Q-use Lucene query syntax to query specified matches
- From _-default query start point 0
- Doc_type-Document Type
- Size-the default value of the specified number of queries is 10.
- Field-specified fields are separated by commas (,).
- Sort-sorting field: asc/desc
- Body-use Query DSL
- Scroll-rolling Query
Statistical Query# The syntax is roughly the same as that of search, but only the statistical value is output.
In[52]: es.count(index='logstash-2015.08.21', q='http_status_code:500')Out[52]:{u'_shards':{u'failed':0, u'successful':5, u'total':5}, u'count':17042}
Knowledge Extension
# Initialize the scrollpage = es.search( index ='yourIndex', doc_type ='yourType', scroll ='2m', search_type ='scan', size =1000, body ={ # Your query's body})sid = page['_scroll_id']scroll_size = page['hits']['total']# Start scrollingwhile(scroll_size >0): print "Scrolling..." page = es.scroll(scroll_id = sid, scroll ='2m') # Update the scroll ID sid = page['_scroll_id'] # Get the number of results that we returned in the last scroll scroll_size = len(page['hits']['hits']) print "scroll size: "+ str(scroll_size) # Do something with the obtained pageIn the preceding demo, data is retrieved at a time. After the data is obtained, the latest data is not obtained. What should we do if we want to get the latest data after rolling? A statistical value, such as total: 5, is generated during scrolling. After jumping out of the loop, we can use the _ from parameter to locate the data after 5 starts to scroll.
Range filter query rangeGt:> greater than lt: <less than gte: >=greater than or equal to lte: <= less than or equal
"range":{ "money":{ "gt":20, "lt":40 }}
-
Bool combination filterMust: All clauses must match AND be the same as AND. Must_not: All clauses must NOT match and NOT. Shocould: at least one clause matches with OR.
{ "bool":{ "must":[], "should":[], "must_not":[], }}Term Filter
{ "terms":{ "money":20 }}
- Multiple matching conditions are allowed for the complex terms version.
{ "terms":{ "money": [20,30] }}Regular Expression Query
{ "regexp": { "http_status_code": "5.*" }}Match Query
{ "match":{ "email":"123456@qq.com" }}
- Multi_match multi-field search
{ "multi_match":{ "query":"11", "fields":["Tr","Tq"] }}Demo
- Obtain data from the last hour
{'query': {'filtered': {'filter': {'range': {'@timestamp':{'gt':'now-1h'}} } } }}
-
{ "query":{ "filtered":{ "query":{"match":{"http_status_code":500}}, "filter":{"term":{"server_name":"vip03"}} } }}
- Terms Facet single field statistics
{'facets': {'stat': {'terms': {'field':'http_status_code', 'order':'count', 'size':50} } }, 'size':0}
- Count multiple fields at a time
{'facets': {'cip': {'terms': {'fields':['client_ip']}}, 'status_facets':{'terms':{'fields':['http_status_code'], 'order':'term', 'size':50}}}, 'query':{'query_string':{'query':'*'}}, 'size':0}
- Count multiple fields together
{'facets': {'tag': {'terms': {'fields':['http_status_code','client_ip'], 'size':10 } } }, 'query': {'match_all':{}}, 'size':0}
Data assemblyThe following is a demo of the kibana homepage, which is used to count the number of logs in a period of time.
{"Facets": {"0": {"date_histogram": {"field": "@ timestamp", "interval": "5 m"}, "facet_filter ": {"fquery": {"query": {"filtered": {"query": {"query_string": {"query": "*" }}, "filter ": {"bool": {"must": [{"range": {"@ timestamp": {'gt ': 'Now-1h '}}}, {"exists": {"field": "http_status_code.raw" }}, # ------------- ------- # matching conditions] }}}}, "size": 0}
If you want to add matching conditions, add filtering conditions in the above code identification section, follow the code format below
{"query": { "query_string": {"query": "backend_name:baidu.com"} }},
Here, we will introduce the Query dsl api later.