Caffe conversion of picture data to Lmdb and data set mean values

Transfer from website:

http://blog.csdn.net/muyiyushan/article/details/70578077

1. Preparing the Data

Using the Dog/cat dataset, separate the train and Val folders under the training project root as the location where the training data and validation data are saved. Each of the train and Val folders has two folders: Dogs and cats, respectively, to save pictures of dog and cat. Dog and cat have 1000 training images and 400 test images respectively.

Write a Python script file, Traverse train and Val two folders, generate Train.txt and Val.txt, respectively, and save the relative path and label of the picture. Similar to the following format, but it is best to disrupt the path order of cat and dog, not start with cat, and then all dog.

/cats/cat.0.jpg 0/cats/cat.1.jpg 0/cats/cat.10.jpg 0/cats/cat.100.jpg 0/cats/cat.101.jpg 0/cats/cat.102.jpg 0/cats/ Cat.103.jpg 0/cats/cat.104.jpg 0

　　

2. Data preprocessing2.1 Converting data formats

Use Convert_imageset.exe to convert the original image to Caffe direct-read format: Lmdb or LEVELDB. The default is Lmdb, and if you want to generate LEVELDB, add the –BACKEND=LEVELDB parameter. Create a Convert_image.bat file with the following command.

set glog_logtostderr=1e:\code\caffe\ Caffe-master\caffe-master\build\x64\release\convert_imageset.exe--resize_width=256--resize_height=256 E:\Code\ Caffe\caffe-master\caffe-master\data\dogcat\train E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\train.txt E:\code\caffe\caffe-master\caffe-master\data\dogcat\trainldbe:\code\caffe\caffe-master\caffe-master\build\x64\ Release\convert_imageset.exe--resize_width=256--resize_height=256 E:\Code\Caffe\caffe-master\caffe-master\data\ Dogcat\val\ E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\val.txt E:\Code\Caffe\caffe-master\caffe-master\ Data\dogcat\valldbpause  

parameter Description :
1. E:\Code\Caffe\caffe-master\caffe-master\Build\x64\Release\convert_imageset.exe is the execution program path.
2.–resize_width=256–resize_height=256 re-Specifies the size of the generated image.
3. E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\train is the location where the training image is stored.
4. E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\train.txt the index file of the training image.
5. E:\CODE\CAFFE\CAFFE-MASTER\CAFFE-MASTER\DATA\DOGCAT\TRAINLDB generated lmdb format file path.

The test image is the same.

Note : The path of 3 and 4 in Train.txt specify that the path is connected to the absolute path of the image, otherwise the picture is not found)
Execute the Convert_image.bat file to generate the Lmdb file and save it in the specified path, respectively.

2.3 Calculating the image mean

Set up the Compute_mean.bat and execute the content as follows

E:\Code\Caffe\caffe-master\caffe-master\Build\x64\Release\compute_image_mean.exe E:\Code\Caffe\caffe-master\ Caffe-master\data\dogcat\trainldb E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\train_mean.binaryproto

parameter Description
1. E:\Code\Caffe\caffe-master\caffe-master\Build\x64\Release\compute_image_mean.exe is the execution file that calculates the mean.
2. E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\trainldb the Lmdb file path that was generated in the previous step.
3. E:\Code\Caffe\caffe-master\caffe-master\data\dogCat\train_mean.binaryproto generated mean file, suffix is. Binaryproto.

3. Modify the configuration file

Create a new Train_val folder in the project root directory and copy the configuration files from. \caffe-master\models\bvlc_reference_caffenet to Train_val. including Train_val.prototxt and Solver.prototxt.

Modify the Mean_file path in train_val.prototxt (use the same mean file for training and testing) and Data_source path to set the corresponding batch_size. Set the num_output of the last layer Fc8 to the corresponding number of output categories, which is 2.

The solver.prototxt is set as follows.

NET: "E:/code/caffe/caffe-master/caffe-master/data/dogcat/train_val/train_val.prototxt" Test_iter:16test_ Interval:50base_lr:0.01lr_policy: "Step" Gamma:0.1stepsize:2000display:20max_iter:10000momentum:0.9weight_decay: 0.0005snapshot:500snapshot_prefix: "E:/code/caffe/caffe-master/caffe-master/data/dogcat/caffenet_finetune" Solver_mode:cpu

　　

NET parameter is the train_val.prototxt you just set.
Test_iter is the number of batch used for the test, test_iter*testbatch_size=test images.
Test_interval: The number of iterations of the test.
Max_iter: Number of iterations.
...

4. Start Training

Establish Caffe.bat training model. and save the log file.

5. Visual Training Process

Use the log file to visualize parameters such as accuracy and loss.

6. Fine tuning

1-5 completed the training of a model from the beginning, we can also train a model from the beginning. But this takes a lot of time, and if the training data is not enough, the training accuracy is not high. Based on the idea of migration learning, you can use your own data to finetune with imagenet pre-trained models, thus reducing the training time, the effect is better. The parameter initialization method is actually different.

Download the pre-trained model.

Modify Train_val.prototxt and solver.prototxt accordingly. Train_val.prototxt configuration of the data file path, the last layer of the name of the change, set the corresponding num_output, increase weights and bias lr_mult (training from the beginning, to speed up).

Reduce the initial learning rate in solver.prototxt.

Start training.

Caffe conversion of picture data to Lmdb and data set mean (RPM)