distance de hamming

Through the collection system, we collect a large amount of text data, but there are a lot of duplicate data in the text that affects our analysis of the results. Before analysis, we need to remove duplication of the Data. How can we select and

I remember someone asked me before, the page to go to the weight algorithm, I have to say the cosine vector similarity match, but if it is billions of levels of the page to weight it? This is bad, because every two pages need to calculate the vector

Some sites on the internet often have mirror sites (mirror), which are the same as the contents of two Web sites, but the corresponding domain names are different. This causes repeated crawls of the same web crawler multiple times. To avoid this,

What is Simhash?Simhash is a fingerprint generation algorithm or fingerprint extraction algorithm that was mentioned in the paper "Detecting near-duplicates for Web crawling" published by Google in 2007. Google is widely used in billions of pages to

From: http://leoncom.org /? P = 650607  On the way to eat Huludao a few days ago, Fei Ge gave a detailed explanation of his high efficiency in comparing Text Similarity experiments with Google's simhash method. He came back to read the original

When the first look at the beauty of Mathematics, the book mentions this algorithm, at that time did not do related to work, no specific impression. A year ago when the interview when the other people mentioned this algorithm, know that Simhash can

Collected fast one months of information, although not fully understand, but the first slowly write it, perhaps there is a train of thought.The biggest benefit of open source is that it gives the author a sense of shame about dirty smelly code.When

1. IntroductionThe crawler collects a lot of text data, how to carry out the weight? You can use the text to calculate the MD5, and then compare it with the MD5 collection that has been crawled, but there is a problem with a slightly different text

A few days ago to eat gourd head of the road, big fly elder brother to explain in detail he in comparison text similarity experiment to Google's Simhash method efficient marvel, come back deliberately to find the original to read.SimhashThe

Common Data Mining & machine learning knowledge (points)Basis (Basic):MSE (meansquare error mean square), LMS (Least meansquare min-squared), LSM (Least square Methods least squares), MLE (Maximum Likelihoodestimation Maximum likelihood estimation),