Newton Iteration Method for Logistic Regression

Next, the last article:Http://blog.csdn.net/acdreamers/article/details/27365941

 

In the previous articleLogistic RegressionThe problem is that we have written out its maximum likelihood function, and now we want to find the maximum likelihood estimation. Therefore

Then the function calculates the partial derivative and obtains an equation, that is

 

 

We only need to solve all the problems based on this equation, but this is not an easy task.Logistic RegressionThe maximum similarity is obtained.

However, we have also said in the problem of finding the Extreme Value of a multivariate function that the point whose derivative is equal to zero may be the maximum, minimum, or non-extreme value. So it depends on one

CallHessian Matrix. For details, see:Http://blog.csdn.net/acdreamers/article/details/29594175

 

PleaseHessian MatrixFirst, we need to find the second-order partial direction.

 

 

SetHessian Matrix. So there are

 

 

So we can see that all the feature values of the matrix are smaller than zero.Hessian MatrixIs negative, that is

Multivariate functions have local maximum values, which also meets the initial maximum likelihood estimation requirements.Hessian MatrixDescribes the local curvature of a multivariate function.

The iterative form is as follows (solving the equation)

 

 

At the beginning, we will select a vertex as the iteration start point. Sometimes the selection of this vertex is critical because the Newton iteration method obtains the local optimal solution, as shown in figure

If the function only has one zero point, the selection of this point is irrelevant. However, if there are multiple local optimal solutions, it is generally to find

Zero point nearby. ForLogistic RegressionThe local optimal solution is not necessarily the global optimal solution. We can obtain the optimal solution randomly multiple times.

 

For the above equations, we also use the Newton iteration method to solve them.

 

 

The above is symmetric negative, and now the result is a linear equation. Due to symmetric negativeCholeskyBreak down.

 

CholeskyThe decomposition principles will be explained in detail later.

 

In this way, the expressions are iterated until the specified precision converges to the local optimal value. Of course, the global optimal value is acceptable.

Multiple Random Initial values can be optimized.