Large data processing Method Bloom filter

Bushi filter is a kind of random data structure with high space efficiency, its implementation method mainly includes a bit array, which can be implemented by Bitset in C + + and K hash function. The algorithm is based on the principle that when an element is added to a set, the element is input to the K-hash function, and the element is mapped to the K-point of the array in place, and the points are set to 1. When you want to find out whether an element is in the collection, as long as the element is the input of the K-hash function, and then see if the K-point mapped to is 1, if it is all 1, then the element (possibly) in the collection, if there is a 0, the element is not in the collection. This is the basic idea of the algorithm.

Collection S = {x1, x2, x3,... xn} The process of setting the bit array when adding elements:

　

Said to be all 1, the element may exist because the algorithm has an error rate, that is, a nonexistent element may be judged to exist in this collection. To minimize the error rate, you need to know how to determine the size of the bit array m and the number of hash functions based on the number of input elements N. The error rate is minimized when the number of hash functions is k= (LN2) * (m/n). In cases where the error rate is not greater than E, M must be at least equal to N*LG (1/e) to represent a collection of any n elements. But M should also be larger, because the bit array is also guaranteed to be at least half 0, then M should >=NLG (1/e) *lge is probably NLG (1/e) 1.44 times times (LG represents 2 logarithm).

That is: bit array size M >= NLG (1/e) *lge (E is error rate) = NLG (1/e) 1.44 times times (LG denotes 2 logarithm)

Number of hash functions k = (LN2) * (m/n)

So bloom filter is more suitable for applications that can tolerate errors. It saves a lot of storage space with low error rates.

Use range: Pinyin error detection and database system; It can be used to implement data dictionaries, data weighing, or intersection of sets.

Bloom filter maps the elements in the collection into the array, with K (the number of hashes of the hash function) and whether all 1 indicate whether the element is in this set. The disadvantage is that once an element is added to the collection, it cannot be deleted, or the corresponding bit of the element affects other elements.  CBF is an improvement, it changes each bit to a counter, from the support delete operation, that is, each deletion of a keyword, the corresponding k bits (counter) minus one. SBF relates the counter to the number of occurrences of the element, minimizing the frequency of the occurrence of the element in the counter.

Large data processing Method Bloom filter