Paper notes-augmented Lagrange Multiplier Method for Recovery of Low-rank matrices

Thesis title: The augmented Lagrange Multiplier Method for Exact Recovery of corrupted Low-rank matrices

Abstract

1.Robust PCA problem: Recovering a low-rank matrix with anunknown fraction of it entries being Arbitraril Y corrupted.

The RPCA problem is a convex optimization problem: Minimizes a combination of the nuclear Norm and the L1-norm

2. This paper presents a better and faster algorithm for solving RPCA problems: the augmented Lagrange multipliers (ALM) augmented Lagrangian multiplier method is used to solve the problem.

It is 5 times times faster than the previously used APD (accelerated proximal gradient), with improved accuracy and reduced memory requirements.

3.ALM is also used in tensor completion

4.Matlab Code: http://perception.csl.illinois.edu/matrix-rank/home.html

Introduction

1.PCA assumes the given high-dimensional data lie near a much lower-dimensional linear subspace

The derivation of 2.PCA is quite many, the common is the angle of variance, here is another angle:

D is m*n the original data, A is a low rank matrix, E is their difference, R is far less than min (m,n) as the target subspace dimension, and the F-norm in the objective function corresponds to the hypothesis that the data is independently distributed Gaussian noise .

The problem can be achieved by the SVD of D, the pre-r-column vector, and the projection.

3. When the data by the additive Gaussian noise corrosion, the PCA has a good effect, in the case of small noise amplitude , the effect is good.

If a is arbitrarily corroded (that is, e arbitrarily large), the PCA recovered by a can be arbitrarily poor.

4.RPCA:

[1] When E is sparse enough (relative to a), by solving the following convex optimization problem, you can accurately recover a:

The first item is the nuclear norm of a (the sum of its singular values), the second is the L1 norm (the sum of the absolute value of the matrix entries), Λis a Positive weighting parameter

RPCA is also good for large errors or outliers,gross corruption, and the application of background modeling is as follows:

5. (2) Re-conversion to a common convex optimization problem, with any interior point Solver solution, recommended CVX Toolbox [2]

But the inner point method converges quickly on the small matrix, but the matrix is very slow in 70*70, and the slow is due to the interior point method rely on second-order information of the objective function.

Iterative thresholding (IT) method Solution:

For the l1-norm in the objective function: there are [3,4,5,6] and so on;

[7] for nuclear norm in the target function.

[1] ([1] The same name has two versions of a used it, a used [APG]) with it (combined with [3] and [7]); But the convergence is slow, m=800, to run for 8 hours.

[8] A total of 2 algorithms are proposed:

The first one is a accelerated proximal gradient (APG) algorithm applied to the primal, and which is a direct applic Ation of the Fista framework introduced by [4], coupled with a fast continuation technique (Similar techniques has been APS Plied to the matrix completion problem by [9].); The second one is agradient-ascent algorithm applied to the dual of the problem (2). From simulations with matrices of dimension up to M = $, both methods is at least times faster than the iterative T Hresholding method

6. This paper uses ALM to do matrix recovery.  Ealm q-linear convergence Speed While the APG are in theory only sub-linear. Ialm required numbers of the partial svds is significantly less, at least five times faster than APG. And its precision are also higher.

The number of Non-zeros in E computed by Ialm are much more accurate than so by Apg,which often leave many smallnon Z Ero terms in E.

Previous algorithms for Matrix recovery1.the iterative thresholding approach [1]

[1] using IT to solve (2) The Relaxation convex optimization:

Τis a large positive scalar after the two items added to the target function is small, using Lagrange multipliers:

Then the IT approach updates A E Y iteratively:

1. Fix Y, minimize L (a,e,y) update A and E

2. Update Y with A+e=d restrictions (derivative of Y, gradient descent)

Soft-thresholding (shrinkage) operator:ε>0

by [7,3] know: here is very important, many articles are used, a solution to the kernel norm, a solution to the L1 norm.

USVT is the SVD of W

for (4) A, E is optimized, the inner product and the F-norm are combined into the F-norm, and then solved with (6), the concrete does not push, should be able to see.

Although it is very concise and the theory is perfect, but need to iterate many times to converge, Y's learning rate is not easy to determine.

The accelerated proximal Gradient approach

The APG theory [4,10,11],[4] introduces 3 algorithms of PG to APG to APGL.

The following algorithm 4,7 is introduced by the formula (8), attention to A for min, e,y items can be discarded, for E to Min when also the same, (8) in the 2nd and 3rd can and for (6) Two formula in the form of the second item, and then use (6) to solve, the following algorithm is introduced:

The Dual approach

First understand the following duality norm:

Dual norm (Dual norm):

sup{} is the upper bound, basically can be seen as Max, and Max is the difference is that the collection may not have an upper bound value.

The dual norm of a norm is its own.

Several common dual norms:

1. The dual norm of the F-norm is also the F-norm:

2. The dual norm of the L1 norm is the l∞ norm (the absolute maximum value of the element), and the following algorithm is used, noting that the second l∞ norm in Max () of the formula (10) is the L1 norm in the corresponding (2), and λ also corresponds to (2) λ!

3. The dual norm of the spectral norm (The Matrix 2 norm, the maximum singular value induced by the p-2 norm) is the nuclear norm, and the same algorithm is used in the formula (TEN) Max ()

4.

[8] In addition to APG, the second method is to solve the duality of (2) Problem: (FALRTC tensor filling algorithm also uses the dual norm), Note (10) in Max () the first is the spectral norm, because the spectral norm is the p=2 induction norm, so the lower right corner of the Write 2!

The Methods of augmented Lagrange multipliers

ALM Brief: Billagrange multiply the 3rd item in (12)

ALM Benefits:

1. Convergence:

When {U_k} is an increment sequence, F, H is continuously di?erentiable:

{U_k} is bounded:q-linearly

{U_k} is unbounded:super-q-linearly

Various convergence definitions: Https://en.wikipedia.org/wiki/Rate_of_convergence

2. The optimal update step (learning rate) of the 4th line of the algorithm 3 Y_k proves that the u_k is the best, and the parameters are easier to adjust.

The 3.ALM converges to the exact solution, whereas the previous IT and APG are approximate solutions.

Both ALM algorithms for robust PCA (Matrix Recovery)

for (2) using ALM:

Note the points:

Initial value setting of 1.Y

2. Due to (2) Non-smooth, the convergence conclusion of ALM cannot be used directly, but the paper proves that the conclusion is still valid.

3. The faster the u_k increases, the faster the convergence, but the u_k is too large, the 3rd in the algorithm 4, 6 lines for a E of the two sub-problems in IT will become slow, so there is a compromise.

Ealm is given below:

Notice that the 5th line of algorithm 4, while, repeatedly to a E for Min, the result of the 3rd line, the process algorithm performs SVD too many times, so improved:

In fact, do not need while only need one time to be good (feeling is actually the essence is alternative direction method), get Ialm:

The paper proves that the Ialm can converge to the optimal solution, but the convergence rate proves to be more difficult, but the experiment proves that it is still two times convergent, but when the u_k growth is too fast, it can not guarantee the convergence to the optimal solution, so u_k's tuning is a trade-off, see article P9 theorem 2.

An ALM algorithm for Matrix completion

Before doing the matrix recovery, do a matrix fill can also.

Matrix Fill Optimization Issues:

The matrix fill problem is represented in a similar form (2):

Partial augmented Lagrangian function:

For updating E the constraint (second constraint in (15)) should is enforced when minimizing (16)

As before, the algorithm is as follows:

Because of line 7th, Y is always 0 where it needs to be filled.

References

[1] Wright, J., Ganesh, A., Rao, S., Ma, Y.: Robust principal component Analysis:exact recovery of corrupted Low-rank mat Rices via convex optimization. Submitted to Journal of the ACM (2009)

[2] Grant, M., Boyd, S.: Cvx:matlab software for disciplined convex programming (Web page and software). HTTP://STANFORD.E DU/?BOYD/CVX (2009)

[3] Yin, W., Hale, E., Zhang, Y.: fixed-point continuation for L1-minimization:methodologyand convergence. Preprint (2008)

[4] Beck, A., Teboulle, M.: A fast iterative shrinkage-thresholding algorithm for linear inverse problems. SIAM Journal on Imaging Sciences 2 (1), 183{202 (2009)

[5] Yin, W., Osher, S., Goldfarb, D., Darbon, J.: Bregman iterative algorithms for l1-minimization with applications to co mpressed Sensing. SIAM Journal on Imaging Sciences
1 (1), 143{168 (2008)

[6] Cai, J.F, Osher, S., Shen, Z.: linearized Bregman iterations for compressed sensing. Math.comp. 78, 1515{1536 (2009)

[7] Cai, J., Candμes, E., Shen, Z.: A singular value thresholding algorithm for matrix completion. Preprint, Code available athttp://svt.caltech.edu/code.html (2008)

[8] Lin, Z., Ganesh, A., Wright, J., Wu, L., Chen, M., Ma, Y.: Fast convex optimization algorithms for exact recovery of a Corrupted Low-rank matrix. SIAM J. Optimization

[9] Toh, K.C, Yun, S.: An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems. Preprint (2009)

[Ten] Tseng, P.: On accelerated proximal gradient methods for Convex-concave optimization.submitted to SIAM Journal on Opti Mization (2008)
Wright, J., Ganesh, A., Rao, S., Peng, Y., Ma, Y.: Robust principal component Analysis:exact recovery of corrupted L Ow-rank matrices via convex optimization. In:proceedings of advances in Neural information Processing Systems (2009)

Paper notes-augmented Lagrange Multiplier Method for Recovery of Low-rank matrices