Mathematical deduction for maximizing activation of a neuron in a hidden layer

One neuron in the hidden layer is asked to enter the maximum activation input problem, i.e.

The maximum value for this function is:

.......................................... (j=1,2..100) is an unknown amount of 100 equations

Where f is the sigmoid function

We know that the sigmoid function is monotonically increasing, then the most value of the upper function is the maximum value of the function inside. , is a constant, so it is the maximum value of the equation. How to find the maximum value?

Because of the order and so. That is, the maximum value to be asked.

In the matrix theory we have the Cauchy Cauchy inequality:. Because, so that is.

By Cauchy inequality we find an upper bound: and we know when we can reach the upper bound.

It is important to note, however, that this upper bound is related to X, which means that the upper bound is not fixed and increases as the X two norm increases. So we need to fix the upper bound of the equation, which is the fixed x two norm, when we ask for the maximum value of the equation. (Note that x two norm fixing is not to say that the equation of the unknown x is fixed, your x within the 100 XJ can still change, as long as the two norm unchanged can be)

We assume the two norm of the x and then the value of X, (w is the known amount), which is the value of the following equation set x:

The value of x we have obtained is (mental arithmetic can be calculated):

This is the input that takes the maximum value.

Now you may have a question, above is the assumption that x two norm, can be other values, the answer is yes. You can also put the two norm of x equal to 2.

So what we get is that the input that takes the maximum value is multiplied by the elements of the above vector.

In fact, you can set the two norm of X to a few, and finally calculated by the above method is to take the maximum value, just take the maximum size is not the same.

But the increase does not necessarily increase as we know the sigmoid function:

Therefore, the value of more than 5, and will not increase, has remained close to 1. Of course, we make the maximum value is not necessarily taken to 1, as long as the maximum value.

In summary, you just put the input X (Note that the input here and Training Network parameters Training sample does not matter) limited to a two-norm a (two-norm X is not possible to make the maximum, here is required to input x in the second norm is to make the upper bound is unchanged, the upper bound can only be x two norm equals A and x=kw), Then we can find the input that will fetch the maximum value.

Finally you want to display the maximum value of the input x in the form of an image, in fact, you do not have to tangle the two norm of input x is limited to a few, because here you are just showing the image, regardless of the X two norm limit in a few, those different is only the coefficient problem, and on the image is just the image pixel depth problem The layout of the image's light and dark has not changed. So if you ask for a bunch of x which makes the largest, if there is only one value (the X-norm is the largest, and is a multiple of the weight parameter), but if only to display the image, so long as you want to show the x as long as the corresponding weight of n times, the display is the same effect. In the program is also divided by the coefficient maximum, instead of dividing by the corresponding coefficients and.

Finish

Mathematical deduction for maximizing activation of a neuron in a hidden layer