sparsity constraint sparse constraint detailed

sparsity constraint sparse constraint detailedIntroduction: A linear model is a model that we often use, such as:
In the text classification, bag-of-words has P = K features, a total of N = 5K text sample, in the image to blur, image classification, there are p=65k pixel features, n=100 samples; and so on.
We can use linear models to solve these problems, such as linear regression, logistic regression, Cox regression. Because P is far greater than N, regularization must be introduced. Why do we use regularization here? Take the linear regression model we are familiar with as an example: the linear regression problem is actually a least squares problem (a least-squares problem), and the objective function is:

, which is the required parameter, note: Here the description using Convex optimization book is slightly different from the usual, in fact x corresponds to the parameter w we often say
The above formula can be converted into:
So we can get its analytic solution:

It seems very simple, but there is a problem, when we are seeking the optimal solution, that is, when we minimize f (x), if the ax-b is irreversible, then the derivative is 0 and the equation has infinite solution. The fact that we get a derivative of 0 is a set of linear equations, using the knowledge we have learned of linear algebra, that is, the solution of a homogeneous linear equation group, when the number of equations is less than the number of parameters, or a linear equation group corresponding coefficient matrix of the number of rows less than the number of columns, there is an infinite number of solutions. So the p that we mentioned earlier corresponds to the row of the coefficient matrix, the number of samples corresponds to the column of the coefficient matrix, so the optimal solution cannot be obtained at this time. And there have been problems with fitting at this point. In fact, in practice we often encounter this situation: the number of variables (sample dimension) is much larger than the number of samples, the number of samples is too small to estimate all the required coefficients. We can consider it on the basis of two ideas: first, whether so many parameters (features) are related to the results we require, or that some variables (characteristics) have no effect on our results or that the effect is negligible; on the other hand, we can find infinite solutions to the parameter W. The solution space is too large, we can give it a constraint, so that the solution space is greatly reduced, or in the new solution space can find the best solution. The usual workarounds are: (1) Forward stepwise (2) Best-subset (3) Ridge regression (4) Lasso regression where (4) We use the sparse constraints we want to explore, so let's take a look at Las So regression return to what is: in fact, we have added such a constraint to our model:
That is, Lasso is adding a L1 regularization item to the linear model:

To illustrate the sparsity, let's take a look at the method (3) Ridge regression: It adds a constraint to the model:
That is, Ridge is adding a L2 regularization item to the linear model:
Ridge only shrinks the variable (shrinkage), but lasso both chooses the variable and shrinks it, as shown in the following figure:
Look at the figure, where the intersection is solved; looking at the left, the circle is easy to intersect with the corners of the constrained area, at which point the W1 is 0, and for a higher dimension, it is likely that there will be a lot more wi for 0, which also means that we have chosen the variables, resulting in sparsity. For the right, the probability of crossing on the axis is much smaller than the former, and it is difficult to form sparsity.
--------------------------------------------------------------------------------------------------------------- -------------------------------------------------------------------------
Review the history of L1 regularization: First, David L. Donoho and Iain M. Johnstone first used a similar L1 formula in the paper "Wavelet soft Threshold denoising", published in 1994: Paper Address:/HTTP ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=412133
In the next 1995 years, Tibshirani the lasso application to the regression problem, and gave a detailed explanation, the paper address: https://statweb.stanford.edu/~tibs/lasso/lasso.pdf
Since then, Tibshirani has applied lasso to several other models, such as logistics regression.
DONOHO2004, CandesandTao2005 again applies lasso to the compression perception. ---------------------------------------------------------------------------------------------------
It is obvious that eq.1 is a convex function, there is an optimal solution, but notice that the L1 regularization term is not a derivative, that is, the non-conductive convex function to find the optimal solution, how to solve the problem. At present, there are 6 main solutions: (1) Transform the non-smooth problem into a smoothing problem (2) The coordinate descent method (coordinate descend) (3) ADMM algorithm (4) Gradient descent Method (Subgradient) (5) Gradient descent (6) accelerated gradient descent
Let's look at the secondary gradient method, which is defined as follows:
How to understand the sub-gradient? If f (x) is a convex function and is x0 at the point of the first-order Taylor expansion, it is possible to:
If f (x) is not x0 at the same place, we can still get a lower bound of f (x):
Take the following function as an example:

At the X1 point function, the secondary gradient is the same as the derivative, there is only one, at the X2 Point, the function is not conductive, the second gradient has more than one, the secondary gradient value range is a closed interval. Thus we solve the idea is: for the guidance of the place, according to the conventional method of direct derivation solution can be, for the non-conductive place, the use of sub-gradient. The following is the optimal solution to the eq.1 of the objective function:
Regardless of the regularization term, the least squares method mentioned above, we have:
For simplicity's sake, we only consider the case of standard orthogonality, which is:
Substituting eq.3 into eq.2 can be:
Assuming the global optimal solution of J (W), there are two possibilities for considering the J variable:
(1) When gradients exist, we have:
Available by eq.5:
Substituting eq.3 and eq.4 into eq.6:
Sign is a symbolic function, easy to get and the same number (assuming the symbol after the calculation can be proved), so that the eq.8 into eq.7:
In the use of eq.8, and then on both sides of the same multiply, you can get:
Available by eq.10 and eq.9:
Here (2) when the gradient does not exist, instantly, there are:
A property is used here: The point x0 is a global minimum point of the convex function f, when and only. (Note: The above formula does not give a specific iterative solution process). You can get:
by eq.13:
Thus, eq.14 also satisfies the eq.11, so both cases can be represented by eq.11, so the optimal solution of eq.1 can be expressed using eq.11.

Reference: http://blog.csdn.net/lansatiankongxxc/article/details/46386341
http://freemind.pluskid.org/machine-learning/sparsity-and-some-basics-of-l1-regularization/# d20da8b6b2900b1772cb16581253a77032cec97e
Ttps://web.stanford.edu/~hastie/talks/sparsity.pdf