Bloom filter Python

Document directory

• Pybloom 1.0.2

Http://bitworking.org/news/380/bloom-filter-resources

The bloom filter, conceivedBurton H. BloomIn 1970, is a space-efficient probabilistic data structure that is used to testWhether an element is a member of a set. False positives are possible, but false negatives are not. elements can be added to the set,But not removed(Though this can be addressed with a counting filter). The more elements that are added to the set, the larger the probability of false positives.

 

Http://www.google.com.hk/ggblog/googlechinablog/2007/07/bloom-filter_7469.html

In daily life, when designing computer software, we often need to determine whether an element is in a collection. For example, in word processing software, you need to check whether an English word is correctly spelled (that is, whether it is in a known dictionary). In the fbi, whether the name of a suspect is already on the suspect list; whether a website has been accessed in a web crawler; and so on.

The most direct method is to store all the elements in the set in the computer. When a new element is encountered, you can directly compare it with the elements in the set. Generally, a set in a computer is stored as a hash table. Its advantage is fast and accurate, but its disadvantage is that it is a free storage space. The bloom filter only needs to hash the table size from 1/8 to 1/4 to solve the same problem.

Why (not in the original article, I understand), because hash requires a lot of Bucket space (BIT) to avoid conflicts if it is to work ). the advantage of bloom allows conflict, but it increases the number of hash functions to reduce the probability of conflict at the same time, so we can use a smaller space.

In addition, the implementation of the hash table usually uses the pointer array to point to the set element, while the implementation of bloom uses bitarray, because you don't need to get this set element, just know if it has any.

Pybloom 1.0.2

Http://pypi.python.org/pypi/pybloom/1.0.2

>>> B = bloomfilter (capacity = 100000, error_rate = 0.001)
>>> B. Add ("test ")
False
>>> "Test" in B
True