Tutorial on executing an asynchronous SOLR query under the gevent framework of Python

I often need to use Python with SOLR for asynchronous requests to work. There is a block of code blocking the SOLR HTTP request until the first one is complete before the second request is executed, the code is as follows:

Import requests #Search 1solrResp = Requests.get (' Http://mysolr.com/solr/statedecoded/search?q=law ') for Doc in Solrresp.json () [' response '] [' docs ']:  print doc[' catch_line '] #Search 2solrResp = requests.get (' http:// Mysolr.com/solr/statedecoded/search?q=shoplifting ') for doc in Solrresp.json () [' Response ' [' Docs ']:  print doc[' Catch_line ']

(We use the requests library for HTTP requests)

It is good to index the document to SOLR through a script, which in turn can work in parallel. I need to expand my work, so the index bottleneck is SOLR, not the network request.

Unfortunately, Python is not as handy as JavaScript or go when it comes to asynchronous programming. However, the Gevent library can bring us some help. The gevent is a libevent library built on native asynchronous calls (select, poll, etc.), libevent well coordinated with many lower-level asynchronous functions.

Using Gevent is simple, and one of the most tangled is Thegevent.monkey.patch_all (), which fixes a number of standard libraries for better asynchronous collaboration with Gevent. It sounds scary, but I'm not having problems with this patch implementation.

without further ado, here's what you do if you use gevents to parallel SOLR requests:

import requestsfrom gevent import monkeyimport geventmonkey.patch_all () class Searcher (object) : "" "simple wrapper-doing a search and collecting the results" "" Def __init__ (Self, searchurl): Self.searchu RL = Searchurl def search (self): Solrresp = Requests.get (self.searchurl) Self.docs = Solrresp.json () [' response '] ['    Docs '] def searchmultiple (URLs): "" "Gevent to execute the passed in URLs; Dump the results "" "Searchers = [Searcher (URL) for URL in URLs] # Gather A handle for each task handles = [] for Sear Cher in Searchers:handles.append (Gevent.spawn (Searcher.search)) # Block until all work are done Gevent.joinall (Handl ES) # Dump The results for searcher in Searchers:print ' Search results for%s '% Searcher.searchurl for doc in S Earcher.docs:print doc[' catch_line '] searchurls = [' Http://mysolr.com/solr/statedecoded/search?q=law ', ' http:/ /mysolr.com/solr/statedecoded/search?q=shoplifting '] 

Searchmultiple (Searchurls)
The code is added, and the JavaScript code is less concise than the same functionality, but it does the work, the essence of the code is the following lines:

# Gather a handle for each taskhandles = []for searcher in Searchers:  handles.append (Gevent.spawn (Searcher.search)) # Block until all Donegevent.joinall (handles)

We let gevent produce searcher.search, we can operate on the resulting task, and then we can wait for all the resulting tasks to complete, and finally export the results.

It's almost like that. If you have any ideas, please leave us a message. Let us know how we can help you with your SOLR search app.

