Caffe running his own data and training __caffe

This article mainly refers to the steps in the end of the text, I ran it again, most of the same information, some additions and changes.

Caffe provides databases for Mnist, Cifar10, and ImageNet1000, and for Mnist and CIFAR10, data sets are ready to be downloaded and completed by calling code, and ImageNet10 data needs to be downloaded by itself. We can refer to the IMAGENET10 processing to generate the LMDB/LEVELDB data format Caffe needs, and then training and testing. 1. Prepare picture Data

Create a new folder in Data MyDatabase. I have 100 classes, 100 training pictures, 30 evaluations, 60 tests.
Training and validation input are described in Train.txt and Val.txt, and these two TXT files list all filenames and their labels.
Generate the commands that Train.txt and Val.txt can provide with Caffe:
Find-name*.jpeg|cut-d '/'-f2-3train.txt
Note the file path, and then manually make the category label. Similarly, generate Val.txt.

When too many samples, you can write their own instructions batch processing.
Matlab code, generate Train.txt and Val.txt:
Batch_in_txt.m

Clear all
CLC
file = Dir (' train_summer/');
%file = Dir (' val_summer/');
TMP = length (file);
File = File (3:tmp);
fp = fopen (' train.txt ', ' at '); % ' at ' open or create file for reading and writing, append data to end of file
%FP = fopen (' val.txt ', ' at ');

For n = 1:length (file)    
    txt = [File (n). Name ' Num2str (n) ' \ n '];
    fprintf (FP, txt);
End

fclose (FP);

In addition, we also need to change the size of the picture to 256x256, Caffe the command on the home page is:
(http://caffe.berkeleyvision.org/gathered/examples/imagenet.html)
For name in/path/to/imagenet/val/*. JPEG;
Todo
Convert-resize 256x256! The name Name name
Done

You can also change the resize=false in the back of the create_imagenet.sh to true to generate a 256x256 picture.

My own picture size is 1920x1080, with Matlab resize to 960x540, intend to see the effect of first. For training, validation, test set, matlab resize the picture to the 960x540 code:

File = Dir (CD);
TMP = length (file);
File = File (3:tmp);

For n = 1:length (file)
    tmp = imread (file (n). name);
    TMP = IMRESIZE (tmp, 0.5, ' bilinear ');
    Imwrite (tmp, file (n). name);
End

Notice in the Matlab window, I am into the picture folder, and then paste the above code directly to the command line to get resize after the picture.

2. Generate the required Lmdb format

In examples/new Folder MyData, copy create_imagenet.sh under Examples/imagenet folder to the folder, renamed Create_ summer.sh, modify the paths of training and validation, as shown below:


After the modification is complete, run the SH:
./examples/mydata/create_summer.sh

Finally, Summer_train_lmdb and Summer_val_lmdb will be obtained, as shown below:

Episode: I always have an error while running create_summer.sh:/convert_imageset:not found
Later found that it is because I pasted the TOOLS path name followed by a few spaces, resulting in tools/convertimageset can not find ... Because Tools/convert_imageset can't find ... Because the Tools/convert_imageset became build/tools (there are spaces)/convert_imageset. 3. Calculate the mean value of the image

The model needs to subtract the mean from each picture, so we need to get the mean value of the training images, using TOOLS/COMPUTE_IMAGE_MEAN.CPP implementations. This CPP is a good example of how to manipulate multiple components, such as the protocol buffer, LEVELDB, landing, and so on.
Similarly, we copy Examples/imagenet's make_imagenet_mean.sh to Examples/mydata, renamed Make_summer_mean.sh, and modify the path in the file.
4. Define Network

Copy all the files in the models/bvlc_reference_caffenet to the Examples/mydata folder, modify the Train_val.prototxt, mainly modify the file paths of each data tier. The following figure:

If you look carefully at train_val.prototxt, you can see that they are basically similar except for the data source and the last level. In training, we use the SOFTMAX-LOSS layer to compute the loss function and initialize the reverse propagation, while in the verification, we used the precision layer detection precision.

There is also a running protocol solver.prototxt, copied over, changing the first line of the path to our own path net: "Examples/mydata/train_val.prototxt". From the inside we can observe that we will run 256 batches, iterations 4.5 million times (90 phases), every 1000 iterations, we test learning Network validation data, We set the initial learning rate at 0.01, every 100000 (20) iterations to reduce the learning rate, display a message, train the Weight_decay to 0.0005, every 10,000 iterations, we show the current state.
The above is a tutorial, in fact, the above need to take a long time, so we change a little bit
test_iter:1000 refers to the test batches, we have 10 photos, set 10 on it.
test_interval:1000 means that once every 1000 iterations are tested, we change it to 500 tests at a time.
base_lr:0.01 is the basic learning rate, because the amount of data is small, 0.01 will fall too fast, so change to 0.001
Lr_policy: The change of "step" learning rate
Rate of change in gamma:0.1 learning rate
stepsize:100000 reduces learning rates every 100,000 iterations
Display:20 every 20-tier display once
max_iter:450000 Maximum number of iterations,
momentum:0.9 Learn the parameters, do not change
weight_decay:0.0005 Learn the parameters, do not change
SNAPSHOT:10000 10,000 display states per iteration, 2000 times here
Solver_mode:gpu at the end of the line, on behalf of the GPU 5. Training Phase

Copy the train_caffenet.sh in the examples/imagenet to the Examples/mydata, rename it train_mysummer.sh, and modify the path as shown:

Run:./examples/mydata/train_mysummer.sh
6. Data Recovery

Copy the resume_training.sh from the examples/imagenet and run it.

Resources:
(HTTP://WWW.YIDIANZIXUN.COM/0A9NPMRH)