Classification algorithm Evaluation criteria

First, Introduction

There are many classification algorithms, and different classification algorithms use many different variants. Different classification algorithms have different specific, different data sets on the performance of different, we need to choose according to the specific task of the algorithm, how to choose the classification, how to evaluate a classification algorithm, the previous decision tree Introduction, we mainly use the correct rate (accuracy) to evaluate the classification algorithm.

The correct rate is really a very good and intuitive evaluation index, but sometimes the correct rate is high and does not represent an algorithm is good. For example, a certain area of the earthquake prediction, suppose we have a bunch of features as an attribute of the earthquake classification, the category only two: 0: No earthquakes, 1: earthquakes occur. A non-thinking classifier that divides the category into 0 for each test case, then it is likely to achieve a 99% accuracy rate, but when the real earthquake comes, the classifier is unaware that the loss from this classification is huge. Why 99% of the correct rate of the classifier is not what we want, because there is uneven distribution of data, Category 1 of the data is too small, completely wrong sub-category 1 can still achieve a high rate of accuracy but ignore the things we are concerned about. Next, the evaluation index of classification algorithm is introduced in detail.

II. Indicators of Evaluation

1, a few common terms

Here we first introduce a few common model evaluation terms, and now assume that we have only two categories of classification targets, the positive examples (positive) and negative examples (negtive) are:

1) True Positives (TP): is correctly divided into the number of positive cases, that is, the actual case and the classifier is divided into a positive example of the number of instances (sample number);

2) False positives (FP): is incorrectly divided into the number of positive cases, that is, the actual negative example, but the classifier is divided into a positive example of the number of instances;

3) False negatives (FN): The number of negative examples is incorrectly divided, that is, the actual case, but the classifier is divided into negative examples;

4) True negatives (TN): is correctly divided into negative cases, that is, the actual negative case and the classifier is divided into negative examples of the number of instances.

Real Inter - Class Don't	Forecast Category
		Yes	No	Total
	Yes	Tp	Fn	P (The actual yes )
	No	Fp	Tn	N (The actual no )
	Total	P ' (is divided into Yes )	N ' (is divided into no )	P+n

Is the four terms of the confusion matrix, I only know the FP called pseudo-Yang rate, the other how to address the unknown. Note that P=TP+FN is actually a positive example of the number of samples, I have mistakenly thought that the actual sample number should be TP+FP, here just remember that true, false describes whether the classifier is correct, Positive, negative is the classification of the classifier results. If the positive example is counted as 1, the negative example is 1, namely positive=1, Negtive=-1, 1 means false, then the actual class label =TF*PN,TF true or FALSE,PN is positive or negtive. For example true positives (TP) the actual class label =1*1=1 is a positive example, false positives (FP) of the actual class label = (-1) *1=-1 is a negative example, false negatives (FN) of the actual class label = (-1) * (-1) =1 as a positive example, True negatives (TN) is a negative example of the actual class =1* (-1) =-1.

2. Evaluation Index

1) correct rate (accuracy)

The correct rate is our most common evaluation indicator, accuracy = (TP+TN)/(P+n), which is easy to understand, is divided by the number of samples to divide by all the number of samples, usually, the higher the correct rate, the better the classifier;

2) Error Rate

The error rate is the opposite of the correct rate, which describes the proportion of the wrong division of the classifier, error rates = (FP+FN)/(P+n), for an instance, the sub-pair and sub-fault is mutually exclusive event, so accuracy =1-error rate;

3) sensitivity (sensitive)

sensitive = tp/p, which represents the proportion of all positive cases being divided, measures the recognition ability of the classifier to the positive example;

4) Effect degree (specificity)

specificity = tn/n, which represents the proportional proportion of all negative cases, measures the recognition ability of the classifier to the negative case;

5) accuracy (precision)

Precision is the measure of accuracy, representing the proportion of the example that is divided into a positive example, precision=tp/(TP+FP);

6) recall rate (recall)

Recall is a measure of coverage, a number of positive cases are divided into positive cases, recall=tp/(TP+FN) =tp/p=sensitive, you can see the recall rate and sensitivity is the same.

7) Other evaluation indicators

• Computational speed: The time required for classifier training and forecasting;
• Robustness: The ability to deal with missing and outlier values;
• Scalability: The ability to handle large data sets;
• Explanatory: The understandable nature of the classifier's predictive standard, like the rules created by the decision tree, is easy to understand, and a bunch of neural network parameters are poorly understood, and we have to think of it as a black box.

For a specific classifier, it is not possible to improve all of the above mentioned indicators at the same time, of course, if a classifier can be correctly divided into all instances, then the indicators have been optimized, but such a classifier often does not exist. For example, we start with the earthquake prediction, no one can accurately predict the occurrence of earthquakes, but we can tolerate a certain degree of false positives, assuming that 1000 predictions, there are 5 predictions for the discovery of earthquakes, one of the real earthquake, while the other 4 times for false positives, then the correct rate from the original 999/1000=99.9% Down to 996/1000=99.6, but the recall rate rose from 0/1=0% to 1/1=100%, so although the number of false claims of earthquakes, but the real earthquake came, we did not miss, such a classifier is what we want, in a certain correct rate, we require the classification of the recall rate as high as possible.

Reference Link: http://www.cnblogs.com/fengfenggirl/p/classification_evaluate.html

Classification algorithm Evaluation criteria