Leveldb Climbing Road--bloom

First, what is the Bron filter

In the beauty of mathematics, there is a chapter about the Bron filter, the contents are as follows.

In word processing software, whether an English word is spelled correctly; In the FBI, the name of a suspect is on the suspect list; in a web crawler, a URL has been visited, and so on. The most straightforward approach is to have all of the elements in the collection present on the computer, and when you encounter a new element, compare it to the elements in the collection. In general, collections in a computer are stored in a hash table. The benefits are fast and accurate, with the disadvantage of consuming storage space. When the collection is very small, the problem is not obvious, and when the collection is large, the problem of low storage efficiency of the hash table is revealed. If you use a hash table to store your email address, you need 1.6GB of memory per 100 million email addresses. In order to solve the problem of the hash table, a mathematical tool called Bron filter is needed. So, the Bron filter is a data tool. His size only requires a hash table size of 1/8 to 1/4 to solve the same problem.

Therefore, the Bron filter is a mathematical tool.

Second, the principle of Bron filter

The Bron filter is actually a very long binary vector and a series of random mapping functions. We use an example of an email address to illustrate it.

If you need to store 100 million e-mail addresses, first set up a 1,600,000,002 binary (bit), that is, 200 million-byte vectors, and then all of the 1.6 billion bits are zeroed. Then for each email address x, use 8 different random number generator (F1, F2, F3 ..., F8) to generate 8 information fingerprints (F1, F2, F3 ...). Then use a random number generator G to set the binary of all 8 positions to 1. After such processing of all 100 million e-mail addresses, a filter for Bron was created.

When we need to see if a suspicious address Y is in the blacklist, use 8 identical random number generator (F1, F2, F3, ..., F8) to generate 8 messages for this address fingerprint s1, S2, S3, ... S8, then the 8 fingerprints corresponding to the Bron filter 8 bits, respectively, is T1, T2, ...., T8. If Y is in the blacklist, then T1, T2, ...., T8 the corresponding 8 binary number is definitely 1.

Third, the advantages and disadvantages of Bron filter

Advantages: Fast, save space

Disadvantage: There is a certain rate of false recognition

Four, leveldb filter in the cloth

Because there is no actual use of leveldb, so, personally, it is felt here that the LEVELDB filter is faster and more space-saving when searching the database. After the specific use of leveldb, then to understand bloom. Here's a look at Code analysis.

Bloomfilterpolicy is inherited from the Filterpolicy, about Filterpolicy in the later study in detail, this section only discusses bloom.cc.

1. Bloomfilterpolicy class

1.1 Bloomfilterpolicy

constructor, primarily to initialize, and then determine how many hash functions are required

Explicit bloomfilterpolicy (int bits_per_key): Bits_per_key_ (Bits_per_key) {//We intentionally round down to red  UCE probing cost a little bit k_ = static_cast<size_t> (Bits_per_key * 0.69);    0.69 =~ ln (2) if (K_ < 1) k_ = 1;  if (K_ >) k_ = 30; }

1.2 Name

Returns the Bloom filter name

Virtual const char* Name () const {return ' LEVELDB.  BuiltinBloomFilter2 "; }

1.3 Createfilter

Create Bloomfilter,keys is the key that needs to be deposited, N is the number that needs to be deposited, DST is the result of Bloomfilter

Leveldb adds a k_ to the final Bloomfilter, indicating how many hash functions are used, so that at query time you can know directly how many hash functions to use, without having to re-use a variable to record how many hash functions are used.

Virtual void createfilter (CONST&NBSP;SLICE*&NBSP;KEYS,&NBSP;INT&NBSP;N,&NBSP;STD::STRING*&NBSP;DST)  const {    // Compute bloom filter size  (In both  bits and bytes)     size_t bits = n * bits_per_key _;//the total number of bits for all keys that need to be created     // for small n, we can see a  very high false positive rate.  fix it    //  by enforcing a minimum bloom filter length.    if   (bits < 64)  bits = 64;//minimum required 64 bits to save//These two lines are primarily byte-aligned     size_ t bytes =  (bits + 7)  / 8;//8-byte alignment of the occupied memory     bits =  bytes * 8;//total number of digits required     const size_t init_size = dst- >size ();    dst->resize (init_size + bytes, 0);//Set all the bits to 0    dst->push_back (Static_cast<char> (K_));   // remember # of probes in filter,   Deposit The total number of hash functions last     char* array = & (*DST) [init_size];     for  (int i = 0; i < n; i++)  {       // use double-hashing to generate a sequence of hash  values.      // see analysis in [kirsch,mitzenmacher  2006].      uint32_t h = bloomhash (Keys[i]);       const uint32_t delta =  (H&NBSP;&GT;&GT;&NBSP;17)  |  (h &NBSP;&LT;&LT;&NBSP;15);  // rotate right 17 bits       for  (size_t j = 0; j < k_; j++)  {         const uint32_t bitpos = h % bits;//gets the position of the first number in bits//  data into an array  /* &NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;&NBSP;BITPOS/8 compute elements in the first few bytes;           (1 <<  (bitpos % 8)) calculates the element in bytes of the first;           Example: The value of          bitpos is 3,  Then the element should be assigned a value of 1 on the third bit of the first byte. The value of          bitpos is 11, then the element is on the third bit of the second byte, then this should be assigned a value of 1.           Why use the |= operation, because the value on the byte bit may be 1, then the new value is assigned, and the original value needs to be preserved.          */        array[ bitpos/8] |=  (1 <<  (bitpos % 8));         h += delta;      }    }  } 

1.4 Keymaymatch

Query whether there is a function, key is required to query, Bloom_filter is the need to use a comparison of the filter

Virtual bool keymaymatch (const slice& key, const slice& bloom_ Filter)  const {    const size_t len = bloom_filter.size ();     if  (len < 2)  return false;    const  char* array = bloom_filter.data ();     const size_t bits  =  (len - 1)  * 8;    // Use the encoded  k so that we can read filters generated by    //  bloom filters created using different parameters.    const  size_t k = array[len-1];//here is the number of hash functions saved using the tail of the filter     if  (k > &NBSP;30)  {//Retention short filter       // Reserved for potentially  New encodings for short bloom filters.      // Consider it a  match.      return true;    }     Uint32_t h = bloomhash (Key);    const uint32_t delta =  (h >> 17)  |  (h << 15);   // rotate right 17  bits    for  (size_t j = 0; j < k; j++)  {      const uint32_t bitpos = h % bits;//Find       if  (array[bitpos/8] &  (1 <<  (bitpos  % 8))  == 0)  return false;//Determine if there is a fabric filter in the       h  += delta;    }    return true;  }

The above is leveldb in the main code and analysis of bloom, you can consider, in the future when writing code, if there is a large number of data needs to query, read, you can first through the Bron filter to see if there is, and then read. And the filter is also a mathematical method, from the side of the relationship between mathematics and computer, so, there is time still need to study mathematics in depth.

More sharing, in the Exchange QQ Group: 199546072

Leveldb Climbing Road--bloom