Use Lucene. net

This article only records some simple usage methods for beginners.
The following example uses Lucene. Net 1.9 and can be downloaded from Lucene. net.

1. Basic ApplicationsUsing system;
Using system. Collections. Generic;
Using system. text;
Using Lucene. net;
Using Lucene. net. analysis;
Using Lucene. net. analysis. Standard;
Using Lucene. net. documents;
Using Lucene. net. index;
Using Lucene. net. queryparsers;
Using Lucene. net. search;
Using Lucene. net. store;
Using Lucene. net. util;

Namespace consoleapplication1.lucene
{
Public class deleetest
{
Private const string fieldname = "name ";
Private const string fieldvalue = "value ";

Private directory = new ramdirectory ();
Private analyzer = new standardanalyzer ();

Public deleetest ()
{
}

Private void index ()
{
Indexwriter writer = new indexwriter (directory, analyzer, true );
Writer. maxfieldlength = 1000;

For (INT I = 1; I <= 100; I ++)
{
Document document = new document ();

Document. Add (new field (fieldname, "name" + I, field. Store. Yes, field. Index. un_tokenized ));
Document. Add (new field (fieldvalue, "Hello, world! ", Field. Store. Yes, field. Index. tokenized ));

Writer. adddocument (document );
}

Writer. Optimize ();
Writer. Close ();
}

Private void search ()
{
Query query = queryparser. parse ("name *", fieldname, analyzer );

Indexsearcher searcher = new indexsearcher (directory );

Hits hits = searcher. Search (query );

Console. writeline ("qualified records: {0}; Total number of index database records: {1}", hits. Length (), searcher. Reader. numdocs ());
For (INT I = 0; I {
Int docid = hits. ID (I );
String name = hits. DOC (I). Get (fieldname );
String value = hits. DOC (I). Get (fieldvalue );
Float score = hits. Score (I );

Console. writeline ("{0}: docid: {1}; Name: {2}; Value: {3}; score: {4 }",
I + 1, docid, name, value, score );
}

Searcher. Close ();
}
}
}

In addition to ramdirectory, you can also use fsdirectory. (Note that when the create parameter of fsdirectory. getdirectory is set to true, the existing index library files will be deleted. You can use indexreader. indexexists () to determine the value .)

Open an existing index library from the specified directory.

Private directory = fsdirectory. getdirectory ("C: \ Index", false );

Load the index library into the memory to increase the search speed.

Private directory = new ramdirectory (fsdirectory. getdirectory (@ "C: \ Index", false ));
// Or
// Private directory = new ramdirectory (C: \ Index ");

2. Multi-field search

You can use multifieldqueryparser to specify multiple search fields. Query query = multifieldqueryparser. parse ("name *", new string [] {fieldname, fieldvalue}, analyzer );

Indexreader reader = indexreader. Open (directory );
Indexsearcher searcher = new indexsearcher (Reader );
Hits hits = searcher. Search (query );

3. Multi-condition search

In addition to using queryparser. parse to break down complex search syntaxes, you can also combine multiple queries. Query query1 = new termquery (new term (fieldvalue, "name1"); // word search
Query query2 = new wildcardquery (new term (fieldname, "name *"); // wildcard
// Query query3 = new prefixquery (new term (fieldname, "name1"); // field search field: keyword, automatically added at the end *
// Query query4 = new rangequery (new term (fieldnumber, numbertools. longtostring (11l), new term (fieldnumber, numbertools. longtostring (13l), true); // range search
// Query query5 = new filteredquery (query, filter); // search with filter conditions

Booleanquery query = new booleanquery ();
Query. Add (query1, booleanclause. occur. Must );
Query. Add (query2, booleanclause. occur. Must );

Indexsearcher searcher = new indexsearcher (Reader );
Hits hits = searcher. Search (query );

4. Set Weights

You can add weights (boost) to documents and fields so that they rank top in search results. By default, the search result is sorted by document. Score. The larger the value, the higher the ranking. The default boost value is 1. Score = score * boost

Using the formula above, we can set different weights to influence the ranking.

In the following example, different weights are set based on the VIP level.

Document document = new document ();
Switch (VIP)
{
Case VIP. Gold: Document. setboost (2f); break;
Case VIP. Argentine: Document. setboost (1.5f); break;
}

As long as boost is large enough, a hit result will always rank first, which is the "billing ranking" service of Baidu and other websites. Obviously unfair.

5. Sort

With the construction parameters of sortfield, we can set sorting fields, sorting conditions, and inverted sorting. Sort = new sort (New sortfield (fieldname, sortfield. Doc, false ));

Indexsearcher searcher = new indexsearcher (Reader );
Hits hits = searcher. Search (query, sort );

Sorting has a great impact on search speed. Try not to use multiple sorting conditions.

6. Filter

Filter is used to filter the search results to obtain more accurate results in a smaller range.

For example, we can search for products that are available between-10-1 and-10-30.
For the date and time, we need to convert it to add it to the index database, and it must also be an index field.

// Index
Document. Add (fielddate, datefield. datetostring (date), field. Store. Yes, field. Index. un_tokenized );

//...

// Search
Filter filter = new datefilter (fielddate, datetime. parse ("2005-10-1"), datetime. parse ("2005-10-30 "));
Hits hits = searcher. Search (query, filter );

In addition to the date and time, you can also use integers. For example, the search price is between 100 and ~ Between 200.
Lucene. Net numbertools performs bitwise processing on numbers. If you need floating point numbers, you can refer to the source code.

// Index
Document. Add (new field (fieldnumber, numbertools. longtostring (long) price), field. Store. Yes, field. Index. un_tokenized ));

//...

// Search
Filter filter = new rangefilter (fieldnumber, numbertools. longtostring (100l), numbertools. longtostring (200l), true, true );
Hits hits = searcher. Search (query, filter );

Use query as the filter condition.

Queryfilter filter = new queryfilter (queryparser. parse ("name2", fieldvalue, analyzer ));

We can also use filteredquery for multi-condition filtering. Filter filter = new datefilter (fielddate, datetime. parse ("2005-10-10"), datetime. parse ("2005-10-15 "));
Filter filter2 = new rangefilter (fieldnumber, numbertools. longtostring (11l), numbertools. longtostring (13l), true, true );

Query query = queryparser. parse ("name *", fieldname, analyzer );
Query = new filteredquery (query, filter );
Query = new filteredquery (query, filter2 );

Indexsearcher searcher = new indexsearcher (Reader );
Hits hits = searcher. Search (query );

7. distributed search

We can use multireader or multisearcher to search for multiple index libraries. Multireader reader = new multireader (New indexreader [] {indexreader. Open (@ "C: \ Index"), indexreader. Open (@ "\ Server \ Index ")});
Indexsearcher searcher = new indexsearcher (Reader );
Hits hits = searcher. Search (query );

Or indexsearcher searcher1 = new indexsearcher (reader1 );
Indexsearcher searcher2 = new indexsearcher (reader2 );
Multisearcher searcher = new multisearcher (New searchable [] {searcher1, searcher2 });
Hits hits = searcher. Search (query );

You can also use parallelmultisearcher for multi-thread parallel search.

8. Merge the index database

Merge directory1 into directory2.

Directory directory1 = fsdirectory. getdirectory ("index1", false );
Directory directory2 = fsdirectory. getdirectory ("index2", false );

Indexwriter writer = new indexwriter (directory2, analyzer, false );
Writer. addindexes (new directory [] {directory });
Console. writeline (writer. doccount ());
Writer. Close ();

9. display the search syntax string

We have combined many search conditions and may want to see what the equivalent search syntax string is.

Booleanquery query = new booleanquery ();
Query. Add (query1, true, false );
Query. Add (query2, true, false );
//...

Console. writeline ("Syntax: {0}", query. tostring ());

Output:
Syntax: + (Name: name * value: name *) + number: [cannot exceed limit B to exceed limit D]

It's that simple.

10. Operate the index database

Delete (soft Delete, only the delete tag is added. Call indexwriter. Optimize () and delete it .)

Indexreader reader = indexreader. Open (directory );

// Delete the document with the specified serial number (docid.
Reader. Delete (123 );

// Delete the document containing the specified term.
Reader. Delete (new term (fieldvalue, "hello "));

// Restore soft deletion.
Reader. undeleteall ();

Reader. Close ();

Incremental update (you only need to set the create parameter to false to add new data to the existing index database .)

Directory directory = fsdirectory. getdirectory ("Index", false );
Indexwriter writer = new indexwriter (directory, analyzer, false );
Writer. adddocument (doc1 );
Writer. adddocument (doc2 );
Writer. Optimize ();
Writer. Close ();

11. Optimization

When you add indexes to fsdirectory in batches, increasing the merge factor and minmergedocs helps improve performance and reduce the index time. Indexwriter writer = new indexwriter (directory, analyzer, true );

Writer. maxfieldlength = 1000; // maximum field length
Writer. mergefactor = 1000;
Writer. minmergedocs = 1000;

For (INT I = 0; I <10000; I ++)
{
// Add external entes...
}

Writer. Optimize ();
Writer. Close ();

Parameter description

Transferred from in-depth Lucene index mechanism

Using Lucene, you can make full use of the hardware resources of the machine in the index creation project to improve the index efficiency. When you need to index a large number of files, you will notice that the bottleneck of the index process is the process of writing index files to the disk. To solve this problem, Lucene holds a buffer in the memory. But how do we control the Lucene buffer? Fortunately, Lucene's indexwriter class provides three parameters to adjust the buffer size and the frequency of writing index files to the disk.

1. Merge factor)

This parameter determines how many documents can be stored in an index block of Lucene and how often the index block on the disk is merged into a large index block. For example, if the merge factor value is 10, when the number of documents in the memory reaches 10, all documents must be written to a new index block on the disk. In addition, if the index block on the disk reaches 10, the 10 index blocks will be merged into a new index block. The default value of this parameter is 10. This value is very inappropriate if the number of documents to be indexed is very large. For batch indexes, assigning a large value for this parameter will produce better index results.

2. Min Number of merged documents (minmergedocs)

This parameter also affects the index performance. It determines the number of documents in the memory at least to write them back to the disk. The default value of this parameter is 10. If you have enough memory, setting this value as large as possible will significantly improve the index performance.

3. maxmergedocs)

This parameter determines the maximum number of documents in an index block. The default value is integer. max_value: setting this parameter to a relatively large value can improve index efficiency and search speed. Because the default value of this parameter is the maximum value of an integer, we generally do not need to change this parameter.

------------------- Confused split line -----------------------------

Lucene resources:

1. Lucene Official Website
2. Apache Lucene
3. Lucene FAQ
4. Lucene. net
5. Lucene API (Java)
6. dotlucene
7. Luke-Lucene index toolbox
8. nutch
9. Lucene. com. cn China
10. Compass
11. Lucene in practice, Part 1: initial knowledge of Lucene
12. In-depth Lucene index mechanism