Notes on Training recurrent networks online without backtrackinglink:http://arxiv.org/abs/1507.07680
Summary
This paper suggests a method (Nobacktrack) for training recurrent neural networks on an online method, i.e. without have to Do Backprop through time. One of the understanding the method is that it applies the forward method for automatic differentiation, but since it requ Ires Maintaining a large Jacobian matrix (nb of hidden units times nb. of parameters), they propose a-to Stochastic (but unbiased!) estimate of the that matrix. Moreover, the method is improved by using a Kalman filtering on this estimate, effectively smoothing the estimate over time.
My cents
online training of Rnns is a big, unsolved problem. The current approach people use are to truncate backprop to only a few steps in the past, which are more of a heuristic.&nbs P This paper makes progress towards a more principled approach. I really like the "Rank-one trick" of Equation 7, really cute! And it is quite central to this method too, so good job on connecting those dots! the authors present ing preliminary, and indeed they do not compare with truncated backprop. I really hope they do in a future version of this work. also, I don't think I buy their argument that the "Theo Ry of stochastic gradient descent applies ". Here ' s the reason. So the method tracks the Jacobian of the hidden state wrt the parameter, which they note G (t). It is update to G (t+1), using a recursion which is based on the chain rule. However, between computing G (t) and G (t+1), a gradient step is performed during training. This means. G (t) is now slightly stale, and coRresponds to the gradient with respect-to-old value of the parameters, not the current value. As far as I-understand, this implies that G (t+1) (more specifically, its stochastic estimate as proposed in this paper) is N ' t unbiased anymore. So, unless I ' M missing something (which I might!), I don ' t think we can invoke the theory of SGD as they suggest. &nb Sp But frankly, which last issue seems pretty unavoidable in the online setting. I suspect this would never be solved, and the future of the future of the. Robust to this issue (or develop new theory that shows it's isn ' t one).  SO Overall, kudos to the authors, and I ' m reall Y looking forward to read more about where this is the goes!
the Fine Print: I Write these notes sometimes hastily, and thus they might not all perfectly reflect what ' s in the paper. They is mostly meant to provide a first impression of the paper ' s topic, contribution and achievements. If your appetite is wet, I ' d recommend you dive in the paper and check for yourself. Oh, and do let me know if you think I got things wrong:-)
Notes on Training recurrent networks online without backtracking
