Caffe Deep Learning Advanced Cifar-10 Classification Task (top)

Preface

CIFAR-10 datasets are a common data set in the field of deep learning. The Cifar-10 consists of 60000 32*32 RGB color images, all of which include aircraft, cars, birds, fur, deer, dogs, frogs, horses, boats and trucks in 10 categories. 50000 training, 10000 tests. is often used as a classification task to evaluate the merits and demerits of deep learning frameworks and models. More well-known models such as Alexnet, NIN, ResNet, etc. have all evaluated their performance on the CIFAR-10 data set. It also has a sister-level dataset Cifar-100, which, as the name implies, contains 100 categories, and the data is more complex. A description of the CIFAR data set and the download of the data are available on the website. It is because the CIFAR-10 data set is small, the category is clear, easy to access, training is simple, and the model can be easily referenced, so as a beginner of deep learning as a step-by-step content, again suitable (the premise of this article is the use of Caffe and related parameters have a certain understanding). First Model

After the data is taken, the first thing is to get a model to run and find the feeling, then the first model to choose what. I've been thinking about it for a long time, LeNet. AlexNet. or a vgg. ResNet. In fact, no matter what model to choose, I recommend that you first carefully read the relevant papers, understand its principles, to do the experiment will be handy, and the experimental part of the paper will also provide the relevant parameters of the experiment, but also to save the time for everyone to tune the parameter. You can refer to my previous blog (Vgg translations, Vgg notes, ResNet translations, ResNet notes) for Vgg and ResNet's paper translations and notes. When you know the approximate idea of a model (any can), move your mouse and keyboard, on GitHub can be downloaded to the model of Daniel built a framework, can be used directly. But here I also suggest that, when you get the prototxt from others, read the framework carefully, you can use the Model visualizer Netscope to see the deployment of the model, which is helpful for you to know more about this model. So you can start your first experiment.

I'm using resnet here, why. Because as the winner of the ImageNet2015, ResNet is still one of the most widely used models in the past few years, is also a hot topic of research, extensibility, its advantages undoubtedly, and the most important point, resnet the original word, Only 20 layer of resnet on the Cifar-10 accuracy rate can reach more than 91%, looking at the heart, but also strong reference. So he chose ResNet. GitHub has a lot of ResNet source code, here I recommend: Resnet-on-cifar10, provide detailed source code and the corresponding results. Here I used the 20-storey resnet he provided to carry out the experiment.

Model, data ready to start training, specific parameters set here on the not detailed said, we can see the paper, the entire training process is smooth, the final test accuracy reached 91.65%, because the training curve was not preserved, here to steal a map (thank the source of the provider), we forgive:

The result is basically in line with the results in the paper, but I'm still not satisfied, so I took the next step, data enhancement. Data Enhancement

Data enhancement is a kind of technical means used in deep learning training model, it can improve the diversity of data, and can effectively alleviate the overfitting of model, and can achieve the effect of improving model performance and generalization ability. Familiar with image processing should know that there are many ways to enhance the image, such as rotation, translation, zooming and so on. You will often read about data enhancement in your paper, which is not covered here.

You can use the data enhancement in the outside of the model training, that is, before training is to expand your data in the way of data enhancement, you can also write data enhancement in the Caffe source, similar to the Caffe of the mirror, when the data entered, will randomly do the data you write to enhance. I recommend the second method here, because if you use the outside of the model, first you need a lot of time to process your data, and you also need a lot of space to store your data, which is very detrimental to the use of resources.

Here you can choose your own familiar data enhancement method to expand the data, but to remind you that the addition of data enhancement does not necessarily give the performance of the model gain, for example, observed in the CIFAR-10 data set, test samples and training samples in the direction of the target is the same, Therefore, if you rotate the training sample, it will not improve the performance of the model, and may even play a role in Vice. So before choosing a data enhancement, we recommend that you look at the samples in the dataset before choosing the data enhancement you need to use. This is a pit, a lot of people have stepped on, I am no exception.

After the data enhancement, you will find that the performance of the model has undergone a qualitative leap, the test accuracy rate may have jumped from 91% to 92%, 93% or higher. But we are still not satisfied with this result, how to do. We chose to extend the model. Model Extensions

ResNet is a very extensive model that is very extensible, with extensions on the ResNet model including wide ResNet, ResNet in ResNet, Stochastic Depth ResNet, fractalnet, and so on. Their ideas, as well as the implementation are very good, it is very recommended that you pass these papers over, for everyone to understand the model structure is very helpful, here I also provide you with some paper notes (Wide ResNet, fractalnet). Take wide resnet, in fact, his idea is very simple, is to increase the width of resnet, which is the model of each convolutional layer of the channel number, the principle is easy to understand, the implementation is very simple. We will directly on the basis of the previous ResNet-20 modification, the output of its convolution layer to increase the number of channels twice times, 4 times times and 8 times times, 3 models can be used to compare experiments.

Here we remind you that the first convolutional layer of resnet is not widened, so the number of channels for the first convolutional layer and the first residual block is not the same, and the shortcut connection cannot be made. Then we need to add a convolution layer on the first shortcut to the same Elevation feature map channel number, the increase method can refer to the original ResNet different channel number of residual block between the map shortcut. After modification, you can run directly, if the error is correct according to the reason to modify, until you can run through. (Through this step, we can find that now no longer online download other people's code, you can modify the code to achieve their own needs of the function, is not great. ）

Through such a simple extension to achieve the wide ResNet, training after the test results, twice times the wrns reached more than 95%, and 4 times times and 8 times times the wrns reached more than 96%, is not great.

Through the expansion of the model, the performance of the model has been a qualitative leap, but at the same time it also brings the calculation of the amount of computing resources, parameters and so on, in the actual project, we often hope that on the basis of guaranteeing a certain performance, as far as possible to reduce the number of calculations and parameters, thus the maximum possible use of computational resources We have also made a series of attempts.

(not to be continued)