Viewing linear regression from maximum likelihood

Transferred from: http://blog.csdn.net/ppn029012/article/details/8908104

Author: ppn029012

1. Review of linear regression

In the previous section, when we tried to solve the relationship between "house price and Size", linear regression was used to fit a linear equation so that the linear equations were best matched with the room size data.

So the solution to our problem is that

Take the data as a fact

To match data with a specific model (e.g. linear or nonlinear equation)

This data is treated as God , and let's use models to match them. Data is a fact, when the error is very large, can only show that the model is not good enough, still need to work hard to match our data.
2. See linear regression in a different perspective

Just now the data is the fact, in a different perspective, the data should be a manifestation of the facts. That is, "house price data " should be " housing prices and the size of the relationship" a performance. Now suppose that the relationship between house prices and size in Beijing has been determined.

Room rate = House size *500,

But we don't know, now we got 5 data,

(500, 1), (502, 1), (1510, 3), (1120, 2), (1500, 2). Will find that these 5 data do not match the relationship. This is why this is because the data contains not only the relationship between "house price" and "size", but also probably includes, "house price" and "Old and new", "house" and "direction", "house" and "Community environment" ... And so on, and these factors are likely to be observable, and may not be observed.

So it is possible to predict perfectly and accurately the relationship between the house size and the housing price !! Just find out all the factors that affect the price.

It's impossible to find all the factors that affect house prices!! So we may be reluctant to, just need a recent relationship, so long as the other factors are considered to be some of the size of the house is not related to small noise is good. So

Y is our house price, F (x) is the relationship between house prices and the size of houses, \epsilon is some small noise unrelated to the size of the house, of course, because \epsilon is a random thing, we can use the random variable E to represent it,

3. Maximum Likelihood

Anyway, now that we have a string of x, Y, we can try to find the most probable f (x) to fit the data.

What do you mean, most likely?

If there is a M f (x), then we need to evaluate which model is most likely to produce this string of data D (Y, X). Probability should be expressed in probabilities,

is the f (x) parameter, if the data is independent from the data, there is

The following equation represents the probability that the model produces data x, y

Because X, Y has been determined, now to make the most possible, we can only pass the adjusted value.

For any one data, (xi, Yi), we can calculate

Now to calculate the probability of a model producing data, we just need to know the error between the predicted value of the model and the actual value, and the distribution of the noise random variable E. the process of solving the maximum likelihood problem

Here, the problem can be solved, that is, for the existing data d (x, y) and any one of the parameters of F (x.), to find the best parameters we need,

Select a model f (x), and initialize its parameters

Estimate the distribution of the noise random variable e (e.g. uniform distribution, Gaussian distribution ...) to get likelihood expression

Calculate the likelihood function and adjust the likelihood to achieve maximum

The method of adjustment can be used as described in the previous chapter "derivative Descent Method", of course, you can also directly find the extremum point (derivative of 0) to obtain its maximum minimum value.

The likelihood function varies depending on the selection of the model F (x) and the choice of the noise random variable e . Let me show you how maximum likelihood is associated with the previous two-bit regression (linear regression (Linear Regression) and categorical regression (Logistic Regression)).
4. Maximum likelihood variable linear regression

At this time, the model I choose f (x) = ax + B, noise random variable e a normal distribution N (0,2).

To make likelihood the biggest, you just need to minimize the good. Oh, ah ah. Is this formula familiar? Is this the cost function of the preceding linear regression ? The original linear regression is only a special case of maximum likelihood!
5. Maximum likelihood and classification

At this time, the model I choose f (x) =, the random noise variable distribution is no longer Gaussian distribution, is an extremely complex distribution. But fortunately, we can get the likelihood expression, because

Unify this equation,

So there,

Finally, we can see that this likelihood function becomes the negative of the cost function in categorical regression. So maximizing likelihood is equivalent to minimizing the cost function in categorical regression.

The above two types of problems, linear regression and categorical regression can be deduced from the maximum likelihood estimation method, which shows that the maximum likelihood estimation method is a more universal method to describe the model matching.