Lucene-customscorequery custom sorting

Custom sorting (sorting of non-single-value fields and non-text relevance) is required in some scenarios. Besides rewriting collect and weight, you can use customscorequery.

Scenario: Sort tags by the number of tags in the tag field (the more tags, the higher the score)

public class CustomScoreTest {    public static void main(String[] args) throws IOException {        Directory dir = new RAMDirectory();        Analyzer analyzer = new WhitespaceAnalyzer(Version.LUCENE_4_9);        IndexWriterConfig conf = new IndexWriterConfig(Version.LUCENE_4_9, analyzer);        IndexWriter writer = new IndexWriter(dir, conf);        Document doc1 = new Document();        FieldType type1 = new FieldType();        type1.setIndexed(true);        type1.setStored(true);        type1.setStoreTermVectors(true);        Field field1 = new Field("f1", "fox", type1);        doc1.add(field1);        Field field2 = new Field("tag", "fox1 fox2 fox3 ", type1);        doc1.add(field2);        writer.addDocument(doc1);        //        field1.setStringValue("fox");        field2.setStringValue("fox1");        doc1 = new Document();        doc1.add(field1);        doc1.add(field2);        writer.addDocument(doc1);        //        field1.setStringValue("fox");        field2.setStringValue("fox1 fox2 fox3 fox4");        doc1 = new Document();        doc1.add(field1);        doc1.add(field2);        writer.addDocument(doc1);        //        writer.commit();        //        IndexSearcher searcher = new IndexSearcher(DirectoryReader.open(dir));        Query query = new MatchAllDocsQuery();        CountingQuery customQuery = new CountingQuery(query);        int n = 10;        TopDocs tds = searcher.search(query, n);        ScoreDoc[] sds = tds.scoreDocs;        for (ScoreDoc sd : sds) {            System.out.println(searcher.doc(sd.doc));        }    }}

Test results:

Document<stored,indexed,tokenized,termVector<f1:fox> stored,indexed,tokenized,termVector<tag:fox1 fox2 fox3 >>Document<stored,indexed,tokenized,termVector<f1:fox> stored,indexed,tokenized,termVector<tag:fox1>>Document<stored,indexed,tokenized,termVector<f1:fox> stored,indexed,tokenized,termVector<tag:fox1 fox2 fox3 fox4>>

 

Custom score:

public class CountingQuery extends CustomScoreQuery {    public CountingQuery(Query subQuery) {        super(subQuery);    }    protected CustomScoreProvider getCustomScoreProvider(AtomicReaderContext context) throws IOException {        return new CountingQueryScoreProvider(context, "tag");    }}

 

public class CountingQueryScoreProvider extends CustomScoreProvider {    String field;    public CountingQueryScoreProvider(AtomicReaderContext context) {        super(context);    }    public CountingQueryScoreProvider(AtomicReaderContext context, String field) {        super(context);        this.field = field;    }    public float customScore(int doc, float subQueryScore, float valSrcScores[]) throws IOException {        IndexReader r = context.reader();        Terms tv = r.getTermVector(doc, field);        TermsEnum termsEnum = null;        int numTerms = 0;        if (tv != null) {            termsEnum = tv.iterator(termsEnum);            while ((termsEnum.next()) != null) {                numTerms++;            }        }        return (float) (numTerms);    }}

 

Usage:

Countingquery customquery = new countingquery (query );

The test results are as follows:

Document<stored,indexed,tokenized,termVector<f1:fox> stored,indexed,tokenized,termVector<tag:fox1 fox2 fox3 fox4>>Document<stored,indexed,tokenized,termVector<f1:fox> stored,indexed,tokenized,termVector<tag:fox1 fox2 fox3 >>Document<stored,indexed,tokenized,termVector<f1:fox> stored,indexed,tokenized,termVector<tag:fox1>>

 

//-----------------------

Weight/score/similarity

Collector

 

References

Http://opensourceconnections.com/blog/2014/03/12/using-customscorequery-for-custom-solrlucene-scoring/

Snapshot:

One item stands out on that list as a little low-level but not quite as bad as building a custom Lucene query: customscorequery. when you implement your own Lucene query, you're taking control of two things:

Matching-What sort ents shocould be added in the search results
Scoring-What score shocould be assigned to a document (and therefore what order shoshould they appear in)
Frequently you'll find that existing Lucene queries will do fine with matching but you 'd like to take control of just the scoring/ordering. that's what customscorequery gives you-the ability to wrap another Lucene query and rescore it.

For example, let's say you're re searching our favorite dataset-scifi stackexchange, a q & A site dedicated to nerdy scifi and fantasy questions. the posts on the site are tagged by topic: "star-trek", "star-wars", etc. lets say for whatever reason we want to search for a tag and order it by the number of tags such that questions with the most tags are sorted to the top.

In this example, a simple termquery cocould be sufficient for matching. To identify the questions tagged Star Trek with Lucene, you 'd simply run the following query:

Term termToSearch = new Term(“tag”, “star-trek”);TermQuery starTrekQ = new TermQuery(termToSearch);searcher.search(starTrekQ);

 

If we examined the order of the results of this search, they 'd come back in the default TF-IDF order.

With customscorequery, we can intercept the matching query and assign a new score to it thus altering the order.

Step 1 override customscorequery to create our own scom scored Query Class:

(Note this code can be found in this GitHub repo)

public class CountingQuery extends CustomScoreQuery {public CountingQuery(Query subQuery) {super(subQuery);}protected CustomScoreProvider getCustomScoreProvider(AtomicReaderContext context) throws IOException {return new CountingQueryScoreProvider("tag", context);}}

 

Notice the code for "getcustomscoreprovider" this is where we'll return an object that will provide the magic we need. it takes an atomicreadercontext, which is a wrapper on an indexreader. if you recall, this hooks us in to all the data structures available for scoring a document: Lucene's inverted index, term vectors, etc.

Step 2 create customscoreprovider

The real magic happens in customscoreprovider. This is where we'll rescore the document. I'll show you a boilerplate implementation before we dig in

public class CountingQueryScoreProvider extends CustomScoreProvider {String _field;public CountingQueryScoreProvider(String field, AtomicReaderContext context) {super(context);_field = field;}public float customScore(int doc, float subQueryScore, float valSrcScores[]) throws IOException {return (float)(1.0f);}}

 

This customscoreprovider rescores all documents by returning a 1.0 score for them, thus negating their default relevancy sort order.

Step 3 implement rescoring

With termvectors on for our field, we can simply loop through and count the tokens in the field:

public float customScore(int doc, float subQueryScore, float valSrcScores[]) throws IOException {IndexReader r = context.reader();Terms tv = r.getTermVector(doc, _field);TermsEnum termsEnum = null;termsEnum = tv.iterator(termsEnum);int numTerms = 0;while((termsEnum.next()) != null) {numTerms++;}return (float)(numTerms);}

 

And there you have it, we 've overridden the score of another query! If you 'd like to see a full example, see my "Lucene-query-Example" repository that has this as well as my custom Lucene query examples.

Customscorequery vs a full-blown custom Query

Creating a customscorequery is a much easier thing to do than implementing a complete query. there are a lot of INS-and-outs for implementing a full-blown Lucene query. so when creating a custom matching behavior isn't important and you're only rescoring another Lucene query, customscorequery is a clear winner. considering how frequently Lucene based technologies are used for "fuzzy" analytics, I can see using customscorequery a lot when the regular tricks don't pan out.

 

Lucene-customscorequery custom sorting