Loss Function-1

Http://www.ics.uci.edu /~ Dramanan/teaching/ics273a_winter08/lectures/lecture14.pdf

1. Loss Function

   The loss function can be seen as the loss term and regularization term)

1.1 Loss Term

• Gold Standard (ideal case)
• Hinge (SVM, soft margin)
• Log (logistic regression, cross entropy error)
• Squared loss (linear regression)
• Exponential loss (Boosting)

??

Gold Standard, also known as 0-1 loss, records the number of classification errors

Hinge Losshttp: // en.wikipedia.org/wiki/Hinge_loss

For an intended output?T? = ± 1? And a classifier score?Y, The hinge loss of the prediction?Y? Is defined

Note that?Y? Shocould be the "raw" output of the classifier's demo-function, not the predicted class label. E. g., in linear SVMs ,?

It can be seen that when?T? And?Y? Have the same sign (meaning?Y? Predicts the right class) and?

, The hinge loss?

, But when they have opposite sign ,?

Increases linearly?Y? (One-sided error ).

??

From http://en.wikipedia.org/wiki/Hinge_loss>

Plot of hinge loss (blue) vs. zero-one loss (misclassification, green:Y? <0)?T? = 1? And variable?Y. Note that the hinge loss penalizes predictions?Y? <1, corresponding to the notion of a margin in a support vector machine.

??

From http://en.wikipedia.org/wiki/Hinge_loss>

??

??

In Pegasos: Primal Estimated sub-GrAdient SOlver for SVM

Here we take the first part as the normalization part, and the second part as the error part. Pay attention to the comparison between ng's Courseware on svm

Rule-based

Normalization

??

Log Loss

Ng courseware 1, first talk about linear regression and then lead to the least square error, then the probability angle Gaussian distribution to explain the minimum error.

Then, let's talk about logical regression. MLE is used to bring out the optimization goal, which is to maximize the probability of training data.

??

??

Maximize the following log likelihood functions

This is precisely to minimize cross entropy!

??

Http://en.wikipedia.org/wiki/Cross_entropy

Http://www.cnblogs.com/rocketfan/p/3350450.html information theory, relationship between cross entropy and KL divergence

??

Cross entropy can be used to define loss function in machine learning and optimization. The true probability?

? Is the true label, and the given distribution?

? Is the predicted value of the current model.

More specifically, let us consider? Logistic regression, which (in its most basic guise) deals with classifying a given set of data points into two possible classes generically labeled?

? And?

. The logistic regression model thus predicts an output?

, Given an input vector?

. The probability is modeled using thelogistic function?

. Namely, the probability of finding the output?

? Is given

Where the vector of weights?

? Is learned through some appropriate algorithm such? Gradient descent. Similarly, the conjugate probability of finding the output?

? Is simply given

The true (observed) probabilities can be expressed similarly?

? And?

.

??

Having set up our notation ,?

? And?

, We can use cross entropy to get a measure for similarity?

? And?

:

The typical loss function that one uses in logistic regression is computed by taking the average of all cross-entropies in the sample. For specifically, suppose we have?

? Samples with each sample labeled?

. The loss function is then given:

Where?

,?

? The logistic function as before.

??

The logistic loss is sometimes called cross-entropy loss. it's also known as log loss (In this case, the binary label is often denoted by {-1, + 1 }). [1]

??

From http://en.wikipedia.org/wiki/Cross_entropy>

??

??

Therefore, it is exactly the same as ng's conclusion from the MLE point of view! The difference is the outermost negative number.

That is, the objective function of logistic regression optimization is cross entropy.

??

Squared loss

??

Exponential loss

The exponential error is usually used in boosting. The index error is always greater than 0, but the smaller the result error is, the larger the result error is.

??

??

Loss Function-1