2016.3.28 How to determine the initial parameters

How to determine the initial parameters

there are so many parameters, such as the learning rateYita, the size of the punishmentLambda, as wellMini-Batchand when the internet stops.

a more macroscopic way of thinking is to verify the idea, there is no need to use all the data set, but to use a part of it, validation is effective, and then to use a larger data set, and in order to verify the idea, you can properly reduce the depth of the network layer, and do more frequent Validation , is actually equivalent to doing a quick small experiment, can train the network faster, see the effect of the idea.

So the idea is very effective at the time of the experiment. Remember: Use small networks for small sample testing.

So what are the appropriate parameters for learning?

Learning rate: The bigger the better after the order of magnitude is determined.

early stop : Because learning the correct rate of learning in the concussion, it is difficult to determine exactly in which round stop, so use for example n The accuracy of the wheel does not rise to determine the stop. A possible problem is that it may be at a certain stage, learning to a platform, a little more time to continue to decline, but this problem is not considered so much in consideration, because the probability is relatively small, or feel good enough, and then modify the n But this is really an effective way to stop training even though.

Lambda: This is to punishWtoo large, determined the learning rate after the relevant debugging, find a better value, in turn, adjust the learning rateLambda. This alternates with each other to achieve a better result. In fact, the learning rate can be seen as the step length, and the gradient drop can be seen as the direction, I want to walk in some direction long distance, andLambdadecided the direction, according to the original idea, actually in one direction of the value to change the time is slightly reduced, will not change particularly big, so is equivalent, I am not the current point on the adjustment, but slightly reduced some after the adjustment, but the total sense of the strange, because although it is reduced , but is actually equivalent to the re-parameter space to transform a position, and then follow the original location of a learning.

Mini - Batch Size : In order to give full play to the computing power of computer, it can be used to calculate or parallel computing, so that the computational efficiency is much higher, so it can effectively reduce the real time of training in this way.

Here's the question:

1.    Is there a program that can automatically tune the parameters, of course, I'm not talking about enumerations, but I'm going through some kind of a way of interacting with the results of the network output.

2. Lambda can be seen as a hope to reduce the w, after small for the value of the special wonderful is not very sensitive, but if from the parameter space, is scaled to another point after the adjustment, then what is the relationship between the two points?

2016.3.28 How to determine the initial parameters