Numerical optimization: The basic theory of computation

1. Optimization issues

The most general optimization problem is expressed in this way:

Solve equality Constraints $\boldsymbol{g} (\boldsymbol{x}) =0$ and Inequality Constraints $\boldsymbol{h} (\boldsymbol{x}) \leq 0$ make $\min f (\boldsymbol{x}) $ 's Solution $\boldsymbol{x}$

where $f: \mathbb{r}^n\rightarrow \mathbb{r}$,$\boldsymbol{g}:\mathbb{r}^n\rightarrow \mathbb{r}^m$ (i.e., m equality constraints), $\ Boldsymbol{h}:\mathbb{r}^n\rightarrow \mathbb{r}^k$ (i.e. K inequality constraints)

linear programming Otherwise (as long as there is any one function nonlinearity) is called nonlinear planning . Optimization problem is a very important problem in numerical calculation. As with the nonlinear equations, this seemingly simple problem has no silver-bullet to deal with most problems directly, and the theory is more complicated, and the application of the optimization algorithm is very extensive and important, except to solve nonlinear equations can be used (matlab most commonly used in solving nonlinear equations of the function fsolve is to use the optimization method, see the MATLAB solution equation built-in function in detail), in the AI and machine learning fields also have a very important place. , whether it is Gangzhongko mathematics, or basic least squares problem (also the method used by the usual linear regression), which is essentially a special case of the optimization problem.

The knowledge of advanced mathematics tells us that the necessary non-sufficient condition to obtain the extremum of one-dimensional continuous smooth function is that the value of the point guide is zero . If you try to find the maximum value (Max/min) of a function on a certain interval, the general method is to find out all the extreme points and the endpoints to compare their function values. As with this method, although the general formulation of the optimization problem always takes the "maximum value" as the pursuit goal, in the actual algorithm almost always strives for the extremum as the starting point . As for the maximum value of the total interval, it is always more complex and difficult, and can always construct some functions, its maximum value for the calculation method is almost impossible to find out. The following discussion basically always solves the extremum algorithm.

2. Nature of the problem

2.1 of the solution Uniqueness exists:

If the function $f $ in the n-dimensional bounded closed area $S $ on the continuous, then $f $ on the $S $ must have a global minimum value;

If the function $f $ in the N-dimensional closed area $S $ on the continuous and forward infinity divergence (coercive, i.e. $\lim\limits_{| | x| | \rightarrow \infty}f (\boldsymbol{x}) =+\infty$), then $f $ must have a global minimum on $S $.

The above theorem only guarantees the existence of the minimum value, and does not establish the relationship between the minimum and minimal values. However, for a special kind of function, its minimum value in a certain area must be the minimum value, such function is convex function . A convex function is a function defined on a convex interval that satisfies any two-point line under an abstract function surface, while a convex interval satisfies any two-point line that is still in the interval. a strict convex function defined within a convex interval has a unique minimum value, which is the minimum value of the function at that interval .

2.2 Optimization conditions:

　　First Order optimization condition: one-dimensional function $f: the first-order extremum condition of \mathbb{r}\rightarrow \mathbb{r}$: $f ' (x) =0$; high-dimensional function $f: \mathbb{r}^n\rightarrow \ First-order extremum condition of mathbb{r}$: $\nabla f (\boldsymbol{x}) =0$.

　　second-order optimization condition: one-dimensional function $f: \mathbb{r}\rightarrow \mathbb{r}$: A $f "(x) >0$; a high-dimensional function $f: \mathbb{r}^n\rightarrow \ Second-order extremum condition of mathbb{r}$: $H _f (\boldsymbol{x}) $ positive definite. Where, $ (H_f (\boldsymbol{x})) _{ij}=\frac{\partial^2 f (\boldsymbol{x})}{\partial x_i \partial x_j}$ is called the Hesse (Hessian) matrix, It is the two-time coefficient of the high-dimensional Taylor expansion, which is equivalent to the two-time coefficient (second derivative) of the one-dimensional function Taylor expansion. When the first order optimization condition is satisfied, the $\rightarrow$ of the sea-slug matrix is the minimum point, and the $\rightarrow$ of the sea-slug matrix is the maximal point; the sea-slug matrix is $\rightarrow$ the point is the saddle point; the sea-slug matrix is the singular matrix $\ Rightarrow$ can not determine the type of point, it is theoretically necessary to have higher order optimization conditions.

2.3 Conditions of the problem:

Consider the Taylor series expansion of a one-dimensional function: $f (\hat{x}) \approx f (x^*) +f ' (x^*) h+f ' (x^*) h^2/2$. Near the extremum, $f ' (x^*) =0$, the difference between the minimum value of the function value is the backward error, $|f (x)-F (x^*) |\leq \epsilon$, then there is $h \leq \sqrt{2\epsilon/|f "(x^*) |} $, the precision comparison solves a nonlinear equation that is halved (2n effective digits $\rightarrow$n digits). However, there are many numerical solutions that are equivalent to solving first-order optimization conditions, where the difference between the minimum value of the function value distance should not be used as the backward error, but the absolute value of the first derivative should be used as the backward error.

3. Numerical methods

Similar to the problem of numerical solution equation, the method of numerical optimization is also more iterative method.

3.1 One-dimensional optimization problem

The numerical method of one-dimensional optimization problem includes: Golden section Search method (interval division), Newton Method (First order optimization solution), two consecutive interpolation methods , etc.

3.2 High-dimensional optimization problems

The numerical methods of high dimensional optimization problems include:Nelder-mead simplex method (direct search), steepest descent method, Trust region method , Gauviton method (First order optimization solution), quasi- Newton method (secant update Iteration)-- Bfgs method , conjugate gradient method , etc.

Numerical optimization: The basic theory of computation