--xavier initialization method of deep learning

Original address: http://blog.csdn.net/shuzfan/article/details/51338178

The "Xavier" initialization method is a very effective method of neural network initialization based on a 2010-year paper "Understanding the difficulty of training deep feedforward neural Networks ", unfortunately, until nearly two years, this method has gradually been more people's application and recognition. In order to make the information flow in the network better, the variance of each layer output should be as equal as possible. based on this goal, let's now deduce: which criteria should be met for each layer's weight.

The article assumes a linear activation function and satisfies the derivative of 0 points at 1, i.e.

Now let's start by analyzing a layer of convolution:

Where NI represents the number of inputs.

We have the following variance formula according to the probability statistic knowledge:

In particular, when we assume that both the input and the weights are 0 mean (which is easier to satisfy now that the BN is present), the above formula can be simplified to:

Further assuming that the input x and the weight w are independently distributed, there are:

Thus, in order to ensure that the input is consistent with the output variance, there should be:

For a multilayer network, the variance of a layer can be expressed in cumulative form:

In particular, the reverse propagation of a gradient is similar in form:
In conclusion, in order to ensure that the variance of each layer coincides with the forward propagation and the reverse propagation, the following should be satisfied:

However, the actual number of inputs and outputs are often unequal, so for the sake of balance, the weight variance should be met:

———————————————————————————————————————

———————————————————————————————————————

Those who have learned probability statistics know that the variance of the uniform distribution between [A, b] is:
Therefore, the implementation of the Xavier initialization is the following uniform distribution:

——————————————————————————————————————————

———————————————————————————————————————————

Let's take a look at how this is implemented in Caffe, where the code is in the Include/caffe/filler.hpp file.

<code class= "language-c++ hljs Haskell has-numbering" style= "display:block; padding:0px; Color:inherit; Box-sizing:border-box; font-family: ' Source Code Pro ', monospace;font-size:undefined; White-space:pre; border-radius:0px; Word-wrap:normal; background:transparent; " ><span class= "Hljs-title" style= "Box-sizing:border-box;" >template</span> <typename <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >Dtype</span>> <span class= "Hljs-class" style= "Box-sizing:border-box;" ><span class= "Hljs-keyword" style= "Color:rgb (0, 0, 136); Box-sizing:border-box; " >class</span> <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >XavierFiller</span>: Public <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >filler</span><<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >dtype</span>&gT {public:explicit <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >xavierfiller</span><span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >const</span> <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >FillerParameter</span>& <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >param</span>) </span>: <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >filler</span><<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >dtype</span>><span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >param</span>) </span> {} virtual void <span CLAss= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >fill</span><span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >blob</span><<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >dtype</span>>* <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >blob</span>) </span> {<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102 );" >check</span><span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >blob</span>-><span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); "
    >count</span> () </span>); int fan_in = Blob->count<span class= "Hljs-container" style= "Box-sizing:border-box; " > () </span>/Blob->num<span class= "Hljs-container" style= "Box-sizing:border-box;"
    > () </span>; int fan_out = Blob->count<span class= "Hljs-container" style= "Box-sizing:border-box;" > () </span>/Blob->channels<span class= "Hljs-container" style= "Box-sizing:border-box;"
    > () </span>; <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); "  >Dtype</span> n = fan_in; Default to Fan_in if <span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >this</span>-><span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >filler_param_</span>.<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >variance_norm</span> () </span> = = <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >FillerParameter_VarianceNorm_AVERAGE</span>) {n = <span class= "Hljs-container" style= "Box-sizing:bord Er-box; " > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >fan_in</span> + <span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >fan_out</span>) </span>/<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >dtype</span><span class= "Hljs-container" style= "Box-sizing:border-box;"
    > (2) </span>; } else if <span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >this</span>-><span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >filler_param_</span>.<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >variance_norm</span> () </span> = = <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); "
    >FillerParameter_VarianceNorm_FAN_OUT</span>) {n = fan_out; } <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >Dtype</span> scale = Sqrt<span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >dtype (3) </span>/<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); "
    >n</span>) </span>; Caffe_rng_uniform<<span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >dtype</span>><span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >blob</span>-><span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " &gT;count</span> () </span>,-scale, scale, Blob->mutable_cpu_data<span class= "Hljs-container" style = "Box-sizing:border-box;"
    > () </span>); <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >check_eq</span><span class= "Hljs-container" style= "Box-sizing:border-box;" > (<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102);" >this</span>-><span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >filler_param_</span>.<span class= "Hljs-title" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >sparse</span> () </span>,-1) << "<span class=" Hljs-type "style=" Box-sizing:border-box; Color:rgb (102, 0, 102); " >Sparsity</span> not supported by this <span class= "Hljs-type" style= "Box-sizing:border-box; Color:rgb (102, 0, 102); " >filler</span>. ";}}; </span></code><uL class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li><li style= "Box-sizing:border-box; padding:0px 5px; " >2</li><li style= "Box-sizing:border-box; padding:0px 5px; " >3</li><li style= "Box-sizing:border-box; padding:0px 5px; " >4</li><li style= "Box-sizing:border-box; padding:0px 5px; " >5</li><li style= "Box-sizing:border-box; padding:0px 5px; " >6</li><li style= "Box-sizing:border-box; padding:0px 5px; " >7</li><li style= "Box-sizing:border-box; padding:0px 5px; " >8</li><li style= "Box-sizing:border-box; padding:0px 5px; " >9</li><li style= "Box-sizing:border-box; padding:0px 5px; " >10</li><li style= "Box-sizing:border-box; padding:0px 5px; " >11</li><li style= "Box-sizing:border-box; padding:0px 5px; " >12</li><li style= "Box-sizing:border-box; padding:0px 5px; " >13</li><li style= "Box-sizing:border-box; padding:0px 5px; " >14</li><li style= "Box-sizing:border-box; padding:0px 5px; " >15</li><li style= "Box-sizing:border-box; padding:0px 5px; " >16</li><li style= "Box-sizing:border-box; padding:0px 5px; " >17</li><li style= "Box-sizing:border-box; padding:0px 5px; " >18</li><li style= "Box-sizing:border-box; padding:0px 5px; " >19</li><li style= "Box-sizing:border-box; padding:0px 5px; " >20</li><li style= "Box-sizing:border-box; padding:0px 5px; " >21</li><li style= "Box-sizing:border-box; padding:0px 5px; " >22</li><li style= "Box-sizing:border-box; padding:0px 5px; " >23</li><li style= "Box-sizing:border-box; padding:0px 5px; " >24</li></ul><ul class= "pre-numbering" style= "Box-sizing:border-box; Position:absolute; width:50px; top:0px; left:0px; margin:0px; padding:6px 0px 40px; border-right-width:1px; Border-right-style:solid; Border-right-color:rgb (221, 221, 221); List-style:none; Text-align:right; Background-color:rgb (238, 238, 238); " ><li style= "Box-sizing:border-box; padding:0px 5px; " >1</li><li style= "Box-sizing:border-box; padding:0px 5px; " >2</li><li style= "Box-sizing:border-box; padding:0px 5px; " >3</li><li style= "Box-sizing:border-box; padding:0px 5px; " >4</li><li style= "Box-sizing:border-box; padding:0px 5px; " >5</li><li style= "Box-sizing:border-box; padding:0px 5px; " >6</li><li style= "Box-sizing:border-box; padding:0px 5px; " >7</li><li style= "Box-sizing:border-box; padding:0px 5px; " >8</li><li style= "box-sizing:border-Box padding:0px 5px; " >9</li><li style= "Box-sizing:border-box; padding:0px 5px; " >10</li><li style= "Box-sizing:border-box; padding:0px 5px; " >11</li><li style= "Box-sizing:border-box; padding:0px 5px; " >12</li><li style= "Box-sizing:border-box; padding:0px 5px; " >13</li><li style= "Box-sizing:border-box; padding:0px 5px; " >14</li><li style= "Box-sizing:border-box; padding:0px 5px; " >15</li><li style= "Box-sizing:border-box; padding:0px 5px; " >16</li><li style= "Box-sizing:border-box; padding:0px 5px; " >17</li><li style= "Box-sizing:border-box; padding:0px 5px; " >18</li><li style= "Box-sizing:border-box; padding:0px 5px; " >19</li><li style= "Box-sizing:border-box; padding:0px 5px; " >20</li><li style= "Box-sizing:border-box; padding:0px 5px; " >21</li><li style= "Box-sizing:border-box; padding:0px 5px; " >22</li><li style= "Box-sizing:border-box; padding:0px 5px; " >23</li><li style= "Box-sizing:border-box; padding:0px 5px; " >24</li></ul>

As can be seen above, Caffe's Xavier realization has three kinds of choice

(1) By default, the variance considers only the number of inputs:

(2) Fillerparameter_variancenorm_fan_out, variance only consider output number:

(3) Fillerparameter_variancenorm_average, variance takes into account both input and output number:

I think it's because the forward information is more important because the input is only considered by default.