CS231N Spring Lecture15 Efficient Methods and Hardware for deep learning lecture notes

1. Problems faced by deep learning:

1) The model is getting bigger, difficult to deploy on the mobile side, and it is difficult to update the network.

2) The training time is getting longer, limiting the production of researchers.

3) Too much energy, hardware costs expensive.

Workaround: Joint design algorithms and hardware.

Computing hardware can be divided into two categories: general and private. Generic hardware can also be divided into CPUs and GPUs. Dedicated hardware can be divided into (FPGA and asic,asic more efficient, Google's TPU is ASIC).

2. Algorithms for Efficient Inference

1) pruning, trim off less important neurons and connections. The first step is to use the original network training; The second step is to trim off a portion of the network, and the third step is to continue training the rest of the network. Repeat the second and third steps. Without loss of precision, the network can be reduced to the original one-tenth (continuing to reduce the accuracy will become worse).

2) Weight sharing, weights do not need to be so accurate, you can think of some approximate weights as the same (such as 2.09, 2.12, 1.92, 1.87 can all be regarded as 2). Also on the basis of the original training, in some way to simplify the weight, and then constantly training to adjust the way to simplify the weight. Without loss of precision, the network can be reduced to the original one-eighth.

The first two methods can be used together, the network can be reduced to the original percentage. There is a name deep Compression.

3) quantization, data type. TPU design is mainly to optimize this part.

4) Low Rank approximation the large network into a series of small networks.

5) Binary (two Yuan)/ternary (ternary) Net, the weight is wildly dispersed into ( -1,0,1) three kinds.

6) Winograd transformation, a more efficient method of convolution.

3. Hardware for Efficient Inference

The common purpose of the various hardware in this direction is to reduce the memory read (Minimize memories Access). Hardware needs to be able to predict with a compressed neural network.

EIE (Efficient Inference Engine) (Han et al ISCA 2016): Sparse weights (the weights thrown out as 0), sparse activation values (thrown away as 0 activations), Weight Sharing (4-bit).

4. Algorithms for Efficient Training

1) parallelization. The CPU has developed in accordance with Moore's Law, and the performance of these single threads has increased very slowly over the years, while the number of cores is increasing.

2) Mixed Precision with FP16 and FP32, normal is calculated with 32-bit, but calculate the weight update with 16 bit.

3) Model distillation, with the "soft results" (soft targets) of the well-trained large network as tags provided to the compressed Small network training. This is a paper presented by Hinton, which explains why soft results are better than ground truth.

4) DSD (Dense-sparse-dense Training), first to the original dense network to do pruning, training sparse network, then re-dense out dense network. Han says this is to learn the branches of the tree before learning the leaves. Compared to the original dense network, the re-dense out of higher precision.

5. Hardware for Efficient Training

Computation and memory bandwidth are two factors that affect overall performance.

Han contrasts nvidia Pascal and Volta, blowing a wave of Volta ... Volta has 120 tensor cores and is very good at matrix operations.

　　

CS231N Spring Lecture15 Efficient Methods and Hardware for deep learning lecture notes