Bloom filter for Massive Data Processing

ArticleDirectory

• Preface
• 1. What is Bloom filter?
• Ii. Applicability
• III. Basic principles and key points
• Iv. Expansion
• V. Problem examples

Bloom filter for Massive Data Processing

 

Preface

This blog has compiled ten questions about massive data processing and summarized ten methods. Next, this blog will focus on analyzing the massive data processing methods and rewrite ten questions about massive data processing. If you have any questions, please leave it blank. Thank you.

1. What is Bloom filter?

Bloom FilterIt is a space-efficient random data structure. It uses a bit array to represent a set very concisely and can determine whether an element belongs to the set.Bloom FilterThis kind of efficiency has a certain price: when determining whether an element belongs to a set, it is possible that the elements that do not belong to this set are mistakenly considered to belong to this set (False
Positive). Therefore,Bloom FilterIt is not suitable for applications with zero errors. In applications that can tolerate low error rates,Bloom FilterA small number of errors have been used to save storage space.

Some may want to know the Chinese name of bloom filter. This should not be translated. Whether the translation is appropriate is determined by the princes. In the following sections, if there are many formulas that you may not understand, you just need to know a little.

1.1. Set representation and element Query

Next let's take a lookBloom FilterHow to use a bitarray to represent a set. Initial status,Bloom FilterIsMBit Array, each bit is set0.

To express S = {x1, x2 ,..., Xn} Such N Set of elements, Bloom Filter Use K Independent hash functions ( Hash
Function ), They map each element in the set {1 ,..., M} . For any element X , No I Location of hash function ing Hi (X) It will be set 1 ( 1 ≤ I ≤ K ). Note: If a location is set 1 , So only the first time will work, and the next few times will not have any effect. In, K = 3 And two hash functions select the same position (from the fifth digit on the left, that is, the second "1 ).

 

In JudgmentYIf it belongs to this setYApplicationKHash function, if allHi (y)All1(1≤I≤K), Then we thinkYIs an element in the Set, otherwise it is consideredYIt is not an element in the set. MediumY1It is not an element in the Set (because Y1 points to a "0" bit ).Y2Or belong to this set, or exactly oneFalse
Positive.

1.2 Error Rate Estimation

As we mentioned earlier,Bloom FilterWhen determining whether an element belongs to the set it represents, there will be a certain error rate (False Positive Rate). Next we will estimate the error rate. Before estimation, to simplify the model, we assume thatKN <mEach hash function is completely random. When the setS = {x1,
X2 ,..., Xn}All elements areKMaps hash functionsMIn the bitwise array0The probability is:

Where1/mIndicates the probability of selecting this bit by any hash function (provided that the hash function is completely random ),(1-1/m)Indicates the probability that this digit is not selected for a hash operation. ToSFully mapped to the array, you need to doKNHash. A certain person or0MeaningKNThis probability is (1-1/m)KNPower. LingP
= E-kN/mThis is to simplify the operation. Here we use the approximation commonly used for ecomputing:

 

In the array where p is a bit0Then, P's mathematical expectation E (P) =P'. The error rate (False Positive Rate:

(1-P) is a bit array.1Ratio,(1-P) k indicatesKThe secondary hash is selected.1, That isFalse
Positive Rate. In the above formula, the second step is similar as mentioned above. Now let's look at the first step.P'It's just the mathematical expectation of P. In reality, the value of P may deviate from its mathematical expectation.M. mitzenmacherProof[2]
The ratio of 0 in the bit array is very concentrated in the vicinity of its expected mathematical value. Therefore, the first step of approximation can be established. SeparatePAndP'In the above formula, you must:

ComparedP'AndF', UsePAndFIt is usually more convenient in analysis.

1.3. Optimal Number of Hash Functions

SinceBloom FilterIf multiple hash functions are required to map a set to an array, how many hash functions should be selected to minimize the error rate during element query? There are two mutually exclusive reasons: if there are too many hash functions, the result is obtained when querying an element that does not belong to the set.0However, if the number of hash functions is small0That's all. To obtain the optimal number of hash functions, we need to calculate according to the error rate formula in the previous section.

First usePAndF. NotesF = exp (K ln (1 −e −kn/m )), We orderG
= K ln (1 −e −kn/m), As long as you makeGMinimum,FNaturally, the minimum value is also obtained. BecauseP = e-kN/m, We canGWrite

according to the symmetry rule, it is easy to see that when P = 1/2 , that is, K = ln2 · (m/N) , G gets the minimum value. In this case, the minimum error rate is F equal to (1/2) k ≈
(0.6185) m/N . Note that p is the probability that one of the bits in the array is still 0, so P = 1/2 corresponds to the values of 0 and 1 in a bit array. In other words, to keep the error rate low, it is best to leave half of the Bit Array empty.

One point that needs to be emphasized is that,P = 1/2The minimum error rate does not depend on the approximate value.PAndF. ForF'
= Exp (K ln (1−( 1−1/m) kN )),G' = K ln (1−( 1−1/m) kN),P' = (1−1/m) kN, We canG'Write

According to the symmetry ruleP' = 1/2,G'Obtain the minimum value.

1.4. Size of the Bit Array

Let's take a look at it. When the error rate does not exceed a certain value,Bloom FilterAt least how many bits are required to represent anyNElement Set. Suppose there are a totalUElement. The maximum error rate isBytesNext, we will calculate the number of digits in the bit array.M.

Hypothesis X For any N Set of elements, F (x) Yes X Array. Then for the set X Any element in X , InS
= F (x) Medium Query X Yes, that is S Acceptable X . Apparently, because Bloom Filter Introduced errors, S What is acceptable is not just X It can also Bytes
(U-n) Items False Positive . Therefore, for a definite Bit Array, it can accept the total N + records (u-n) Elements. In N + records (u-n) Element, S Only N So a definite bit array can represent

.MBit Arrays2 mDifferent combinations can be introduced,MBit Array can represent

. Complete SetNA total

, So we needMBit Array can represent allNElement Set, must have

That is:

The premise of approximation in the above formula isNAndЄ uRelatively small, which is often used in actual situations. Based on the above formula, we conclude that the error rate is not greaterBytesIn this case,MAt least equalN
Log2 (1/logs)Can represent anyNElement Set.

 

In the previous section, we calculatedK = ln2 · (m/N)Time Error RateFMinimum.F = (1/2) k = (1/2) mln2/n. Order nowF≤Bytes, Available

This result is a lower bound than the previous one.N log2 (1/logs)BigLog2e≈1.44Times. This indicates that when the number of hash functions is the best, the error rate must not exceedBytes,MAt least the minimum value must be obtained.1.44Times.

1.5 Summary

In computer science, we often encounter the situation of Time-to-space or space-to-time, that is, to achieve the best of one aspect and sacrifice another aspect.Bloom FilterIn addition to the two factors of temporal space, another factor is introduced: error rate. In useBloom FilterWhen determining whether an element belongs to a set, a certain error rate may occur. That is to say, it is possible to mistake elements that do not belong to this set as belonging to this set (False
Positive), But does not mistakenly think that the elements that belong to this set do not belong to this set (False Negative). After increasing the error rate,Bloom FilterA small number of errors are allowed to save a lot of storage space.

SinceBurton bloomIn70ProposalBloom FilterAfter,Bloom
FilterIt is widely used in spelling checks and database systems. In the past 10 or 20 years, with the popularization and development of networks,Bloom FilterEmerging in the network field, variousBloom FilterVariants and new applications are constantly emerging. It is foreseeable that as network applications go deeper, new variants and applications will continue to emerge,Bloom
FilterIt is bound to achieve greater development.

Ii. Applicability

It can be used to implement a data dictionary, to determine the duplication of data, or to obtain the intersection of data sets.

III. Basic principles and key points The principle is very simple, with a Bit Array + k independent hash functions. Set the bit array of the value corresponding to the hash function to 1. If you find that all the corresponding bits of the hash function are 1, obviously, this process does not guarantee that the search result is 100% correct. At the same time, a inserted keyword cannot be deleted, because the bit corresponding to this keyword affects other keywords. Therefore, a simple improvement is the counting bloom filter, which can be deleted by replacing the bitwise array with a counter array.
Another important issue is how to determine the size of the Bit Array m and the number of hash functions based on the number of input elements n. When the number of hash functions is k = (ln2) * (M/N), the error rate is the minimum. If the error rate is not greater than E, m must at least be equal to N * lg (1/E) to represent a set of any n elements. But m should be larger, because at least half of the bit array should be 0, then m should be equal to> = NLG (1/E) * LGE is probably NLG (1/E) 1.44 times (LG represents the base 2 logarithm ).
For example, if the error rate is 0.01, M is 13 times larger than N. In this case, K is about 8.
Note that the unit of M is different from that of N, M is bit, and N is the unit of the number of elements (accurately speaking, the number of different elements ). Generally, the length of a single element is many bits. Therefore, the use of bloom filter memory is usually saved. Iv. Expansion The bloom filter maps the elements in the set to an array. If K (k is the number of Hash Functions) ing bits are all 1, it indicates that the element is not in this set. Counting bloom filter (CBMs) extends each bit in the bit array to a counter, which supports the deletion of elements. Spectral bloom filter (SBF) associates it with the number of occurrences of the Set element. SBF uses the minimum value in counter to represent the occurrence frequency of elements. V. Problem examples Here are two files a and B, each containing 5 billion URLs. Each URL occupies 64 bytes and the memory limit is 4 GB. Let you find the common URLs of files a and B. What if there are three or even n files?
Based on this problem, we calculate the memory usage. 4G = 2 ^ 32 is about 4 billion * 8 is about 34 billion, n = 5 billion, if the error rate is 0.01, 65 billion bits are required. Currently, 34 billion is available, and there are not many differences. This may increase the error rate. In addition, if these URLs correspond one-to-one, you can convert them into IP addresses, which is much simpler. The above content is organized from:

1. Http://blog.csdn.net/jiaomeng/article/details/1495500
2. Http://blog.redfox66.com/post/2010/09/24/mass-data-topic-1-start.aspx.

.