Sparse Coding-dictionary Learning

The most important part of sparse signals is Dictionary. Select? How can we make a reasonable choice?

I. Selection and learning of dictionaries

A basic method for selecting a proper dictionary is to select a pre-defined dictionary, such as a non-sample wavelet, a operable wavelet, a contour boundary, and a curved wave. Recently, many scholars have proposed a dictionary mainly for images, especially for cartoon-like images. It is assumed that segments are smooth and have smooth boundaries.

These dictionaries are accompanied by detailed theoretical analysis to establish sparse representation coefficients for simple signals. In a typical application, the approximate attenuation rate of m items is represented by M optimum non-zero coefficients.

Another method for selecting a dictionary is adjustable-a base or frame is generated under the control of a specific parameter (continuous or discrete. The two most well-known examples are Wavelets and bandelets.

Although pre-defined and adaptive dictionaries usually have fast conversion speeds, they cannot process sparse signals, and these dictionaries are limited to specific types of images and signals and cannot be applied to new ones, any type of signal. Therefore, we need to find a new way to overcome these limitations-through a learning perspective.

The learning-based method first needs to build a training signal set, and then build an empirical learning dictionary, that is, generate potential atoms through empirical data, instead of passing the theoretical model. Such a dictionary can be used as a fixed or redundant dictionary.

Unlike predefined and adaptive dictionaries, learning dictionaries are applicable to any type of signals that meet the definition of sparse fields. However, this will bring a heavier computing burden and can only be applied to low-dimensional signals (at least so far). This is why this method can only be applied to image blocks.

Ii. dictionary Learning Algorithms

Next we will discuss how to use the learning algorithm to construct a. Assume that the training set Yi has been given and is known to be generated by an unknown sparse Field Model M, this training set can deny the unique generation model and select a specific dictionary? This issue was complicated and was proposed by field and oishausen in 1996. Here, we introduce two mechanisms, the first is the optimization direction method (MOD), proposed by engan, the second is K-SVD, proposed by Aharon. 1. The deviation of the hypothesis model is known as the core content of dictionary learning. Our goal is to estimate. Consider the following optimization issues:

2. Mod Algorithm

Obviously, there is no general algorithm that can solve (12.1) and (12.2), which is similar to the reason why we cannot solve (P0. However, there is no reason not to look for a heuristic algorithm and study its performance in special circumstances.

We can look at (12.1) as a nested minimization strategy: In step K, we use Step K-1 to get the dictionary a (k-1 ), solve M instances Yi in the database: this can obtain the matrix, and then we use the least square method to solve the problem:

Here we use the Frobenius norm to evaluate the error (the F-norm of the matrix is the square root of the sum of squares of all elements in the matrix ). Allows you to scale the elements of the obtained dictionary. Increment K and repeat the above cycle until the convergence criterion is met. This block coordinate relaxation algorithm was first proposed by engan, known as the method of optimal direction (MoD). This algorithm described 12.1

 

 

My understanding of the MOD algorithm is as follows:

1. Matching Tracing Algorithm

Before analyzing this algorithm, we will introduce the matching tracing algorithm.

Matching TracingIt was the earliest time-frequency analysis tool to split a known signal into a weighted sum of many called atomic signals and try to find the closest solution to the original signal. An atomic signal is an element in a large atomic library. In mathematical form, you can get the following:

The weight is the atomic signal obtained from dictionary D.

Just as the Fourier series Splits a signal into a series of sine waves, each component has a different coefficient as the weight. The mathematical formula is as follows:

Matching tracing can also be used to separate signals into a series of atoms, and even describe the Fourier series using matching tracing, that is, the set of all sine functions corresponding to the atomic library.

2. Greedy Algorithms

In order to find a group of atomic weighted sums that best match the original signal, it takes too much time to try all combinations of the atomic library. In the paper [1] in 1993 by Mallat S and Zhang Z, a greedy algorithm (Greedy Algorithm) was proposed, which greatly reduced the time for finding an approximate solution. The method is to first find the largest atom in the atomic database with the inner product result of the original signal, find the signal and the inner product result, and then subtract the original signal as the original signal of the next repeated operation, after doing so, you can get a series of atoms until the stop condition is reached.

Assume that Y = D * X, Y is the initial signal, the size is M * P, and X is a matrix of N * P sizes randomly generated. Then the initial dictionary can be obtained.

The detailed algorithm is as follows:

Input: signal:, dictionary.

Output: List of coefficients :.

Initialization:

;

;

Repeat:

Find with maximum inner product;

;

;

;

Until the stop condition is reached, for example :)

In this case, sparse representation is used.

3. Dictionary Update

2, that is, in this formula, substitute it to solve the minimum value of the equation,

Obtain the final dictionary

Contact for the first time, understanding is not in place, please kindly advise Sina Weibo Co., http://blog.sina.com.cn/u/2029100307.

Sparse Coding-dictionary Learning