Tutorial on executing an asynchronous SOLR query under the gevent framework of Python

Source: Internet
Author: User
Tags solr solr query
I often need to use Python with SOLR for asynchronous requests to work. There is a block of code blocking the SOLR HTTP request until the first one is complete before the second request is executed, the code is as follows:

Import requests #Search 1solrResp = Requests.get (' Http://mysolr.com/solr/statedecoded/search?q=law ') for Doc in Solrresp.json () [' response '] [' docs ']:  print doc[' catch_line '] #Search 2solrResp = requests.get (' http:// Mysolr.com/solr/statedecoded/search?q=shoplifting ') for doc in Solrresp.json () [' Response ' [' Docs ']:  print doc[' Catch_line ']

(We use the requests library for HTTP requests)

It is good to index the document to SOLR through a script, which in turn can work in parallel. I need to expand my work, so the index bottleneck is SOLR, not the network request.


Unfortunately, Python is not as handy as JavaScript or go when it comes to asynchronous programming. However, the Gevent library can bring us some help. The gevent is a libevent library built on native asynchronous calls (select, poll, etc.), libevent well coordinated with many lower-level asynchronous functions.

Using Gevent is simple, and one of the most tangled is Thegevent.monkey.patch_all (), which fixes a number of standard libraries for better asynchronous collaboration with Gevent. It sounds scary, but I'm not having problems with this patch implementation.


without further ado, here's what you do if you use gevents to parallel SOLR requests:

import requestsfrom gevent import monkeyimport geventmonkey.patch_all () class Searcher (object) : "" "simple wrapper-doing a search and collecting the results" "" Def __init__ (Self, searchurl): Self.searchu RL = Searchurl def search (self): Solrresp = Requests.get (self.searchurl) Self.docs = Solrresp.json () [' response '] ['    Docs '] def searchmultiple (URLs): "" "Gevent to execute the passed in URLs; Dump the results "" "Searchers = [Searcher (URL) for URL in URLs] # Gather A handle for each task handles = [] for Sear Cher in Searchers:handles.append (Gevent.spawn (Searcher.search)) # Block until all work are done Gevent.joinall (Handl ES) # Dump The results for searcher in Searchers:print ' Search results for%s '% Searcher.searchurl for doc in S Earcher.docs:print doc[' catch_line '] searchurls = [' Http://mysolr.com/solr/statedecoded/search?q=law ', ' http:/ /mysolr.com/solr/statedecoded/search?q=shoplifting '] 


Searchmultiple (Searchurls)
The code is added, and the JavaScript code is less concise than the same functionality, but it does the work, the essence of the code is the following lines:

# Gather a handle for each taskhandles = []for searcher in Searchers:  handles.append (Gevent.spawn (Searcher.search)) # Block until all Donegevent.joinall (handles)

We let gevent produce searcher.search, we can operate on the resulting task, and then we can wait for all the resulting tasks to complete, and finally export the results.

It's almost like that. If you have any ideas, please leave us a message. Let us know how we can help you with your SOLR search app.

  • Related Article

    Contact Us

    The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

    If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

    A Free Trial That Lets You Build Big!

    Start building with 50+ products and up to 12 months usage for Elastic Compute Service

    • Sales Support

      1 on 1 presale consultation

    • After-Sales Support

      24/7 Technical Support 6 Free Tickets per Quarter Faster Response

    • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.