Loss functions (Loss function)-1

Http://www.ics.uci.edu/~dramanan/teaching/ics273a_winter08/lectures/lecture14.pdf

1. Loss Function

   The loss function can be seen as the error section (loss term) + regularization section (regularization term)

1.1 Loss Term

• Gold Standard (ideal case)
• Hinge (SVM, soft margin)
• Log (logistic regression, cross entropy error)
• Squared loss (linear regression)
• Exponential loss (boosting)

Gold Standard, also known as 0-1 loss, records the number of classification errors

Hinge Loss Http://en.wikipedia.org/wiki/Hinge_loss

For a intended output T =±1 and a classifier score y, the hinge loss of the prediction y is D Efined as

Note that y should is the "raw" output of the classifier ' s decision function, not the predicted class label. e.g., in linear SVMs,

It can be seen this when T and y has the same sign (meaning y predicts the right class) and

, the hinge loss

, but when they had opposite sign,

Increases linearly with y (one-sided error).

From

Plot of hinge loss (blue) vs. Zero-one loss (misclassification, green:y < 0) for t = 1 and variable C2>y. Note that the hinge loss penalizes predictions y < 1, corresponding to the notion of a margin in a support VEC Tor Machine.

From

In the pegasos:primal estimated sub-gradient SOlver for SVM paper

Here, the first part is regarded as the regularization part, the second part is the error part, attention to the contrast ng about SVM courseware

No consideration of the rules

Consider the rule of

Log Loss

NG Courseware 1, first speaking linear regression then draw the least squares error, then the probability angle Gaussian distribution to explain the minimum error.

Then the logistic regression, using the MLE to elicit the optimization goal is to make the training data see the most probability of occurrence

Maximize the following log likelihood function

And this is precisely the minimization of cross entropy!

Http://en.wikipedia.org/wiki/Cross_entropy

Http://www.cnblogs.com/rocketfan/p/3350450.html information theory, cross entropy and KL divergence relationship

Cross entropy can is used to define loss function in machine learning and optimization. The true probability

Is the true label, and the given distribution

Is the predicted value of the current model.

More specifically, let us consider logistic regression, which (in it most basic guise) deals with classifying a given set of data points into possible classes generically labelled

and

. The logistic regression model thus predicts an output

, given an input vector

. The probability is modeled using thelogistic function

. Namely, the probability of finding the output

is given by

Where the vector of weights

is learned through some appropriate algorithm such as gradient descent. Similarly, the conjugate probability of finding the output

is simply given by

The true (observed) probabilities can be expressed similarly as

and

.

have set up our notation,

and

, we can use the cross entropy to get a measure for similarity between

and

:

The typical loss function, the one uses in logistic regression are computed by taking the average of all cross-entropies in The sample. For specifically, suppose we have

Samples with all sample labeled by

. The loss function is then given by:

where

, with

The logistic function as before.

The logistic loss is sometimes called cross-entropy loss. It's also known as log loss (the binary label is often denoted by { -1,+1}). [1]

From

So the conclusion with NG from the mle angle is exactly the same! The difference is the outermost minus.

The optimization objective function of logistic regression is the cross entropy

Fix 14.8 This formula courseware should be a bit wrong in the first + should be-, so that the corresponding loss optimization goal is the smaller the better, the greater the MLE corresponds.

Squared loss

Exponential loss

The exponential error is usually used in boosting, and the exponential error is always > 0, but the smaller the error is to ensure that the closer the correct result, the greater the inverse.

Loss functions (Loss function)-1