Relationship between objective functions of cost function of loss function

The loss function and the cost function are the same thing, the objective function is a related but broader concept, and for the objective function the minimization of the constrained condition is the loss function (loss functions)

Give an example to explain:

The functions of the above three graphs are, in turn,. We want to use these three functions to fit the true value of Price,price.

Given that these three functions will output one, this output may be the same as the real value, or it may be different, in order to indicate the good or bad we fit, we use a function to measure the degree of fit , such as:

, this function is called a loss function (loss functions), or a cost function. The smaller the loss function, the better the model fits .

It's not our goal just to make the loss function as small as possible. Not yet.

This time there is also a concept called the risk function (risk functions). The risk function is the expectation of the loss function, because our input and output follow a joint distribution, but this joint distribution is unknown, so it cannot be computed. But we have historical data, that is our training set, about the average loss of the training set is called the empirical risk (empirical risk), that is, so our goal is to minimize, known as the experience of risk minimization .

Are you finished here? Not yet.

If this is the end of the story, then we look at the graph above, which is certainly the most right-most empirical risk function is the smallest, because it is the best for historical data fitting. But it's certainly not the best thing to look at in the picture, because it's over-learning historical data, which results in a bad effect when it's really predicted, a condition called overfitting (over-fitting).

Why this result is caused. Plain English said that is its function is too complex, there are four times, which leads to the following concept, we should not only to minimize the risk of experience, but also to minimize the structural risk . This time defines a function that is specifically designed to measure the complexity of the model , which is also called regularization (regularization) in machine learning. Commonly used are, norm.

In this step we can say that our ultimate optimization function is: the optimization of empirical and structural risks, and this function is called the objective function .

With the above example, the least structural risk is minimal (the simplest model structure), but the empirical risk is greatest (the worst of the historical data fitting); the least experienced risk (best to fit the historical data) is the least, but the structural risk is greatest (the model structure is the most complex), and the good balance is achieved. , which is best used for predicting unknown datasets.

The above understanding is based on Andrew Ng's public class Coursera and Hangyuan Li's "Statistical learning method", if there is a misunderstanding, you are welcome to correct.