K-means: How to choose K (number of cluster)

The current method of determining the number of cluster is to manually determine the number of cluster

Which k is the right one?

In the dataset, we can say it has 4 clusters, or it has 2 clusters, but which is the right answer? In fact, there is no correct answer , the data set to divide the number of cluster is inherently ambiguous, can be 2, 3, 4. This is also part of unsupervised learning because our datasets are not labeled, so there is no clear answer . So it's very difficult to do an algorithm that can automatically find the K value.

Using the elbow rule (elbow method) to select the K value

By drawing the graph of the relationship between K and cost function, as shown on the left, the value of the elbow (when it began to fall very quickly, at the beginning of the elbow) as the K value, k=3.

Not all problems can be solved by drawing elbow diagram, some problems such as the right side of the figure, the position of the elbow is not obvious (the elbow point can be 3,4,5), then cannot determine the K value.

So the elbow diagram is a way to try, but not all the problems can be drawn as good as the left-hand map to determine the K value.

Another way to choose K-value-subsequent purposes (market, etc.)

Through some subsequent purposes, such as some calculation and evaluation of the market. For example, T-shirt's case, is it designed to be 3 sizes or 5 sizes? To determine the value of K by following some of our purposes, we want to T-shirt to accommodate more customers (5 kinds) or want T-shirt to be cheaper (3 kinds).

Summarize

1. Choose K Value now most still choose by hand with your understanding and insight
2. You can try the elbow method to choose the K value, but it is not universal, because in many cases, the elbow point is not clear
3. In most cases , we choose K values for follow-up purposes , see what we do next, and see how our K-values should be chosen.

K-means: How to choose K (number of cluster)