[E2lsh source code analysis] lsh Algorithm Framework Analysis

Location sensitive hashing (lsh) is the most popular Algorithm for approximate Nearest Neighbor Search. It has a solid theoretical basis and has excellent performance in high-dimensional data space. Since the introduction of related knowledge on the internet is relatively simple, I would like to give a summary of the LSH-Related Algorithms and technologies, hoping to provide convenience to interested friends, I also hope that people who are interested can communicate with each other and correct them more.

1. lsh Principle

Nearest Neighbor problem can be defined as follows: a set of n objects is given and a data structure is established. When any object to be queried is given, this data structure returns the most similar DataSet object for the query object. The basic idea of LSH is to use multiple hash functions to map vectors in high-dimensional spaces to low-dimensional spaces, and use the encoding of low-dimensional spaces to represent high-dimensional vectors. By performing multiple hash ing on vector objects, a high-dimensional vector falls into different buckets of different hash tables according to its distribution and its own characteristics. Ideally, it can be considered that a vector object with a close position in a high-dimensional space has a high probability that it will eventually fall into the same bucket, objects that are far away fall into different buckets with a high probability. Therefore, when querying, the query results are obtained by performing the same hash operation on the query vector and combining the query operations in multiple hash tables.

The hash function is used to filter the entire dataset and obtain the distance between the vertex that may meet the query conditions. This avoids distance calculation between the query vertex and the dataset, and improves the query efficiency. This framework can be called a filtering-verification framework.

2. lsh function family definition

To formally describe the nature of the hash function used by lsh, the concept of the LSH function family is defined.

Set S to the data field of the D dimension data point, D to the similarity measurement function, and p and q to any two points in S. The function family H = {H: S-> u} is called (R1, R2, P1, P2)-sensitive. if and only when:

By hashing functions in the LSH function family, we can ensure that the probability of point conflict between the nearest vertex is greater than the probability of point conflict between the distance.

3. General lsh Algorithm Framework

(1) construct an lsh index (hashing)

Define a function family G = {G: S-> u}, where G (v) = (H1 (V ),..., HK (V), hi, H, randomly select l hash functions G1 ,..., GL. Store any V in the data set to the GI (v) bucket, I = 1,..., L.

This is a full-Index algorithm (full-indexing algorithms). It provides a query table for each possible query point. Therefore, it responds to a query point, this algorithm only needs to perform a query (look-up) in a specially constructed hash table ).

(2) lsh Search Algorithm

For a query point Q and the given distance threshold r, search for the bucket G1 (Q ),..., GL (Q), retrieve all vertices V1 ,..., vn serves as the approximate nearest neighbor of candidates. For any VJ, if D (Q, vj) <= r, VJ is returned, where D is the similarity measurement function.

When creating an lsh index, the selected hash function is a Series Function of K lsh functions, in this way, the difference between the probability of point conflict between the distance and the point conflict between the distance is extended, but this also reduces the two values together, therefore, we need to use L hash tables to increase Pn and reduce pF at the same time. Through this construction process, there is a high probability that the point close to the query point Q will be taken out as a candidate to approximate the nearest neighbor and calculate the final distance, the probability of a vertex that is far from the query vertex Q as a candidate to approximate the nearest neighbor is very small, so that the query can be completed within a short period of time.

Reprinted please indicate the author and Article Source: http://blog.csdn.net/jasonding1354/article/details/38227085

References:

1. Chen Yongjian. Research on Key Technologies of content-based large-scale image retrieval [D]. Huazhong University of Science and Technology. 2011

2. Research on similarity search technology based on location-sensitive hash [D]. Nanjing University. 2012