Deep Learning paper Note (2): Neural network regularization via robust weight factorization

Under review as a conference paper at ICLR 2015.

Motivation :

This paper presents a method of regularization, called fame (Factored Mean training). The proposed FAME model aims to apply a similar strategy, yet learns a factorization of each weight matrix such that the F Actors is robust to noise.

The following are the specific practices:

Standard dropout hidden activation:

where r^ (L-1) is dropout noise.

FaMe Hidden activation:

where r^ (l) is also noise, can be dropout or additive/multiplication Gaussian

The only difference is that the weight matrix factorization can be turned into low rank:

Some questions:

To tell the truth, I don't quite understand the motivation of this article. In addition to doing a matrix factorization, basically and standard dropout almost. But standard dropout as a mask noise has the regularization effect, according to the author himself in the abstract inside the statement, do factorization robust to noise. So here's the question:

Q1: Why would robust to noise? Is it because of low rank? Noise corresponds to eigenvalue very small eigenvector, now low rank causes very small eigenvalue to become 0, to some extent has the effect of denoise?

Q2: Why do I need robust to noise? As I said before, dropout noise is just the thing that brings regularization effect, is good. If robust to noise is Q1 inside understand, have denoise effect, then why want to remove good noise?

Q3:v and u The parameters of the two matrices, I am not very clear? The author says the same as standard NN?

Experimental section:

The dataset tested in this article has mnist, CIFAR-10. From the experimental results, the effect is not obvious.

Deep Learning paper Note (2): Neural network regularization via robust weight factorization