Lagrange of Constrained optimization method

The topic here is the constrained optimization method, the Lagrange multiplier method (Lagrange Multiplier) and the KKT condition are very important for solving the constrained optimization problem, and the Lagrange multiplier method can be used to find the optimal value for the optimization problem of equality constraint. If there is an inequality constraint, the KKT condition can be applied to fetch. Of course, the results obtained by these two methods are only necessary, and only if it is a convex function, the sufficient and necessary conditions can be ensured. It is necessary to note that the KKT condition is the generalization of Lagrange multiplier method.

To solve the problem:

\begin{aligned}
&\min_{x \in \mathbb{r}^n} \ f (x) \notag \ \
&S.T. \ \ \ h_i (x) = 0, i =,..., m \ \ \ \ \
& \ \ \ \ \ g_j (x) \le 0,j =,..., n
\end{aligned}

For the optimization problem without any constraints, directly find the derivative of the $f (x) $0 $x $, you can use gradient descent Newton and other methods;

When coupled with an equality constraint $h _i (x) $, the usual method is the Lagrangian multiplier (Lagrange Multiplier), which first introduces the Lagrange multiplier $\alpha \in \mathbb{r}^m$ to construct the Lagrangian function:

\[l (X,\alpha) = f (x) + \sum_i \alpha_i h_i (x) \]

The $x $, $\alpha$ derivation, the derivative is 0, the $x $, the value of $\alpha$ into the $L (X,\alpha) $ to get the best solution:

\[\frac{\partial L (X,\alpha)}{\partial x} = 0 \ \ \ \ \frac{\partial L (x,\alpha)}{\partial \alpha} = 0 \]

Next explain the Lagrange Multiplier of the demolition, first to see an example, for the two-dimensional case of the objective function is $f (x, y) $, in the plane to draw $f (x, y) $ of the contours, as shown in the dotted line, the constraint equation $h (x, y) = 0$ as shown in the Green Line, the target letter Number $f (x, y) $ with constraint $g (x, y) ￥ There are only three cases, intersect, tangent or no intersection, no intersection is certainly not the solution, only the intersection or tangent to get the feasible solution, but the intersection is not the best value, because the intersection means there must be other contours in the line of the inner or outer, The value of the intersection between the new contour line and the target function is larger or smaller, which means that the optimal values can be obtained only when the contours of the curve tangent to the target function.

The above example is the dilemma: the sine qua non of the Lagrange multiplier method is the objective function and the constraint function tangent, then the two normal vectors are parallel, namely

\[\DELTA_XF (x)-\alpha \delta_x h (x) = 0\]

The above formula is the result of the derivation of the Lagrange function $L (X,\alpha) $.

When an inequality constraint is added, the most commonly used method is the KKT condition. Similarly, we write all equations, inequality constraints and F (x) as a formula, also called Lagrangian functions, introducing Lagrange multipliers \alpha, \beta, respectively:

\[L (X,\alpha,\beta) =f (x) + \sum_i \alpha h_i (x) + \sum_i\beta_ig_i (x) \]

By some conditions, we can find the necessary condition of the optimal value, which is called kkt condition.
1. L (A, B, x) the derivative of x is zero;

2. h (x) = 0;

3. A*g (x) = 0;

The candidate optimal values can be obtained after the three equations are taken. The third is very interesting, because G (x) <=0, if you want to satisfy this equation, you must a=0 or g (x) =0. This is a source of many important properties of SVM, such as the concept of support vectors.

Two. Why is the Lagrange multiplier method (Lagrange Multiplier) and kkt conditions able to get the optimal value?

And the KKT condition is necessary to satisfy the optimization problem of the strong duality condition, it can be understood as follows: we ask for Min f (x), L (A, b, x) = f (x) + a*g (x) + b*h (x), a>=0, we can write F (x) as: Max_{a,b} L (A,b,x), Why is it? Because h (x) =0, g (x) <=0, now is the maximum value of L (a,b,x), A*g (x) is <=0, so L (a,b,x) can get the maximum value only if a*g (x) = 0, otherwise, the constraint is not satisfied, so max_{a,b} L (A , b,x) is f (x) when the constraint is met, so our objective function can be written as min_x max_{a,b} L (a,b,x). If using the dual expression: Max_{a,b} min_x L (a,b,x), because our optimization is to satisfy the strong duality (strong duality that the optimal value of the dual equation is equal to the optimal value of the original problem), so that the optimal value of x0, it satisfies the f (x0) = max_{a,b} min _x L (a,b,x) = min_x max_{a,b} L (a,b,x) =f (x0), let's see what happens in the middle of the two equations:

F (x0) = max_{a,b} min_x L (a,b,x) = max_{a,b} min_x f (x) + a*g (x) + b*h (x) = max_{a,b} f (x0) +a*g (x0) +b*h (x0) = f (x0)

You can see that the above-added black place is essentially said min_x f (x) + a*g (x) + b*h (x) in x0 obtained the minimum value, with the Fermat theorem, that is, for the function f (x) + a*g (x) + b*h (x), the derivative to be equal to zero, that is

Gradient of f (x) +a*g (x) gradient + b*h (x) gradient = 0

This is the first condition in the Kkt condition: L (A, B, x) has a derivative of x that is zero.

As previously stated, A*g (x) = 0, at this time the kkt condition of the 3rd condition, of course the known condition H (x) =0 must be satisfied, all the above mentioned, the optimal value of the optimization problem satisfying the strong dual condition must satisfy the KKT condition, that is, three conditions of the above description. The KKT condition can be regarded as the generalization of Lagrange multiplier method.

For unconstrained optimization problems, we can use various gradient methods, inequality constrained optimization

Lagrange Multiplier method

Dual method

KKT conditions

Lagrange of Constrained optimization method