Http://www.ics.uci.edu/~dramanan/teaching/ics273a_winter08/lectures/lecture14.pdf
- Loss Function
The loss function can be seen as the error section (loss term) + regularization section (regularization term)
1.1 Loss Term
- Gold Standard (ideal case)
- Hinge (SVM, soft margin)
- Log (logistic regression, cross entropy error)
- Squared loss (linear regression)
- Exponential loss (boosting)
Gold Standard, also known as 0-1 loss, records the number of classification errors
Hinge Loss Http://en.wikipedia.org/wiki/Hinge_loss
For a intended output T =±1 and a classifier score y, the hinge loss of the prediction y is D Efined as
Note that y should is the "raw" output of the classifier ' s decision function, not the predicted class label. e.g., in linear SVMs,
It can be seen this when T and y has the same sign (meaning y predicts the right class) and
, the hinge loss
, but when they had opposite sign,
Increases linearly with y (one-sided error).
From
Plot of hinge loss (blue) vs. Zero-one loss (misclassification, green:y < 0) for t = 1 and variable C2>y. Note that the hinge loss penalizes predictions y < 1, corresponding to the notion of a margin in a support VEC Tor Machine.
From
In the pegasos:primal estimated sub-gradient SOlver for SVM paper
Here, the first part is regarded as the regularization part, the second part is the error part, attention to the contrast ng about SVM courseware
No consideration of the rules
Consider the rule of
Log Loss
NG Courseware 1, first speaking linear regression then draw the least squares error, then the probability angle Gaussian distribution to explain the minimum error.
Then the logistic regression, using the MLE to elicit the optimization goal is to make the training data see the most probability of occurrence
Maximize the following log likelihood function
And this is precisely the minimization of cross entropy!
Http://en.wikipedia.org/wiki/Cross_entropy
Http://www.cnblogs.com/rocketfan/p/3350450.html information theory, cross entropy and KL divergence relationship
Cross entropy can is used to define loss function in machine learning and optimization. The true probability
Is the true label, and the given distribution
Is the predicted value of the current model.
More specifically, let us consider logistic regression, which (in it most basic guise) deals with classifying a given set of data points into possible classes generically labelled
and
. The logistic regression model thus predicts an output
, given an input vector
. The probability is modeled using thelogistic function
. Namely, the probability of finding the output
is given by
Where the vector of weights
is learned through some appropriate algorithm such as gradient descent. Similarly, the conjugate probability of finding the output
is simply given by
The true (observed) probabilities can be expressed similarly as
and
.
have set up our notation,
and
, we can use the cross entropy to get a measure for similarity between
and
:
The typical loss function, the one uses in logistic regression are computed by taking the average of all cross-entropies in The sample. For specifically, suppose we have
Samples with all sample labeled by
. The loss function is then given by:
where
, with
The logistic function as before.
The logistic loss is sometimes called cross-entropy loss. It's also known as log loss (the binary label is often denoted by { -1,+1}). [1]
From
So the conclusion with NG from the mle angle is exactly the same! The difference is the outermost minus.
The optimization objective function of logistic regression is the cross entropy
Fix 14.8 This formula courseware should be a bit wrong in the first + should be-, so that the corresponding loss optimization goal is the smaller the better, the greater the MLE corresponds.
Squared loss
Exponential loss
The exponential error is usually used in boosting, and the exponential error is always > 0, but the smaller the error is to ensure that the closer the correct result, the greater the inverse.
Loss functions (Loss function)-1