Mahout Series: Canopy algorithm

Canopy algorithm, simple process, easy to implement, it is the algorithm

(1) Set the sample set to S, determine two thresholds T1 and T2, and t1>t2.

(2) To take a sample point P belongs to S, as a canopy, recorded as C, remove p from S.

(3) Calculate the distance of all points to P in s Dist

(4) If the DIST<T1, then the corresponding point to C, as a weak association.

(5) If dist<t2, the corresponding point is moved out of S, as a strong association.

(6) Repeat (2) ~ (5) until S is empty.

The above procedure shows that the point of the dist<t2 belongs to and has only one cluster, and the t2<dist<t1 point may belong to more than one cluster. It can be seen that canopy is a kind of soft clustering.

Canopy has the effect of eliminating isolated points, and K-means is powerless in this respect. After establishing the canopies, you can delete those canopy that contain fewer positions, often those canopy contain isolated points. According to the number of canopy interior points, to determine the number of cluster center K, this effect is relatively good.

The results of canopy clustering are shown below:

Mahout canopy Clustering process is as follows:

(1) to quantify the text.

Before the document is quantified, it is necessary to serialize the document in a directory, called serialization, to write the document content in sequencefile format, key: document title, Value: Document content.

The process of text segmentation, calculation Ngram, feature extraction, generate a dictionary, the calculation of feature weights to generate document Word matrix.

It is worth noting that the process can be replaced by other methods, not limited to the use of mahout, as long as the vector can be generated.