URL deduplication bloom filter algorithm, error and others

URL deduplication bloom filter algorithm, error and others

Fly with me, in the perfect world --- question

I recently read some books, formulas, and algorithms. If I use a word to pick them up, this is an error. Speaking of errors reminds me of a story of my childhood. When I was a child, a friend of his family opened a small store on the second-level road. One day, the customer said to him with the just-bought bulk oil: The weight is not enough. He told the customer that this is an error. The customer is angry that the component is not enough. How is the error? My friend has a way to pull the client to the next door and put the oil on the floor scale. He said, "You see this pot of oil is placed on the floor scale (called the car's Floor, the scale is not displayed. This is an error.

I was very impressed when I was a child.

Now I think of it, I have a better understanding of the error. Different standards, different occasions, and different expectations constitute an error. It can be said that we are always surrounded by errors. Even in terms of management, Fang Xingdong also concluded on his blog that extreme tolerance for errors is an essential quality for managers. I also summarized it myself: medicine is the precise control of fuzzy, and management is the precise fuzzy management.

Let me talk about the relationship between error and efficiency. Errors and efficiency are a pair of conflicting communities. There is a relationship between errors and efficiency in life: "radish is fast, don't wash mud"; "slow work is fine", etc, they all provide a thorough explanation of the dialectical relationship between error and efficiency. For software engineers, especially when writing algorithms, they are familiar with "space for time" and "Time for Space". I would like to add the following sentence: Error for efficiency.

Error change Efficiency

Google blackboard reported an article about how to reuse the URL: convert an average long URL to an average short guid to save space. There is also a common algorithm for URL deduplication: the bloom filter algorithm. The bloom filter algorithm is a fast algorithm used to check whether Element E exists in the Set S. A typical application is to check whether a word exists in the dictionary during spellcheck. There are many types of query algorithms, such as sorting half, B-tree, and hash-code. What are the advantages of bloom filter?
1. The bloom filter does not store the key-value. The bloom filter uses a set of hash algorithms to convert Element E in the Set S into bits;
2. Fast query speed.

We know that hash algorithms generally conflict, and the conflict in the bloom filter results in errors.

Bloom filter is a common algorithm. It now has Java, C ++, C #, Ruby, and other versions of algorithms. Of course there are also many variants to adapt to more needs.
The bloom filter has an error, but the error is within the controllable range. In other words, the annoyance of most errors is that we cannot measure them.

Refer:

Http://blog.likeshow.net/article.asp? Id = 34
Http://blogs.msdn.com/devdev/archive/2006/01/23/516431.aspx
Http://en.wikipedia.org/wiki/Type_ I _and_type_II_errors
Http://en.wikipedia.org/wiki/Bloom_filter
Http://jaspell.sourceforge.net/javadocs/index-files/index-2.html
Http://blogs.msdn.com/devdev/archive/2005/08/17/452827.aspx
Http://www.cc.gatech.edu/fac/Pete.Manolios/research/bloom-filters-verification.html
Http://homepages.inf.ed.ac.uk/s9902505/specksim/doc/index-files/index-2.html
Http://weblogs.asp.net/dfindley/archive/2004/08/19/217485.aspx
Http://www.darkridge.com /~ Jpr5/archive/dads/html/bloomfilter.html
Http://loaf.cantbedone.org/