Machine learning system Design---Error metrics for skewed (skewed) classes

Skewed classes

Skewed classes: The number of a species is much higher (or less than) another class, that is, two extreme cases.

Predicting the classification model of cancer, if there is only a 1% classification error on test set, at first glance is a good result, actually if we predict all Y to be 0 (that is, none is cancer), the classification error is 0.5% (because the ratio of cancer is 0.5%). The error is reduced, is this an improvement to the algorithm? Obviously not. Because the latter method actually does nothing (will all y=0), it is not a good machine learning algorithm. So this error metrics for skewed classes is not possible, then we need to find a suitable for skewed classes above the error metrics.

Precision (Precision) & Recall (recall rate)

Evaluate a classification model with a test set : As shown, a 2*2 classification is performed on the actual and predicted values (assuming this is a 2 classification problem), dividing them into true positives (actually postive, predictions are postive), false positives (predictions are postive, actually is negative), true negative (actually is negative, forecast is also negative), false negative (forecast also negative, actually is positive).

Precision (Precision): true positive (true positive)/predicted number of positive (positive) (predicted y=1) = True positive/(true positive + false positive)

Recall (Recall): true positive (true positive)/actual number of positive (positive) (actually has cancer) = True positive/(true positive + false negative)

So when we use precision and recall to evaluate the algorithm of the previous example (all Y is predicted to be 0), so that its true positive is 0, so that its precision and recall are 0, that is to know that this is not a good algorithm, So we can evaluate whether an algorithm is good by precision each recall. Also gives us a more direct way to evaluate the quality of the model.

We often use Y=1 (not y=0) as the class for which very few (rare) appear (such as the Cancer Class), which is the class to be detected.

Summarize

1. We use precision and recall to measure the quality of the algorithm so that when we encounter skewed classes, even if we set all the predictions to 0 or 1 o'clock, it will not have high Precison and recall
2. When the precision and recall of our algorithms are good, we can say with certainty that our algorithms perform well (even if we encounter skewed classes)
3. When we encounter skewed classes problems, we use the error rate or accuracy ratio of precision and recall to better measure the quality of the algorithm.

Machine learning system Design---Error metrics for skewed (skewed) classes