Caffe Study Notes (10) Convert image data to db (Leveldb/lmdb) file

In the practical application of deep learning, we often use the original data is a picture file, such as Jpg,jpeg,png,tif, and may be the size of the picture is not consistent. The type of data that is often used in Caffe is Lmdb or leveldb, so a problem arises: how can I convert from the original picture file to a db (Leveldb/lmdb) file that can run in Caffe?

In Caffe, the author provides us with such a file:convert_imageset.cpp, which is stored under the Tools folder in the root directory. After compiling, the resulting executable file is placed under buile/tools/, which is used to convert the picture file into a db file that can be used directly in the Caffe framework.

Use format for this file:

Convert_imageset [FLAGS] Rootfolder/listfile db_name

Requires four parameters:

FLAGS: Picture parameter group, followed by detailed description

rootfolder/: Absolute path to picture storage, starting with the Linux system root directory

listfile: A list of picture files, usually a txt file, one picture at a line

db_name: The final generated DB file storage directory

If the image has already been downloaded to the local computer, we first need to create a list of pictures, saved as txt

This article takes the picture of the Caffe program as an example, to explain, the picture directory is example/images/, two pictures, one for cat.jpg, the other for the Fish_bike.jpg, which represents two categories.

We create an SH script file and call the Linux command to generate a picture list:

# sudo vi examples/images/create_filelist.sh

Edit this file, enter the following code and save

#/usr/bin/env SHdata=examples/Imagesecho"Create train.txt ..."RM-RF $DATA/Train.txtfind $DATA-name *cat.jpg | Cut-d'/'-f3 | Sed"s/$/1/">> $DATA/Train.txtfind $DATA-name *bike.jpg | Cut-d'/'-f3 | Sed"s/$/2/">> $DATA/Tmp.txtcat $DATA/tmp.txt>> $DATA/TRAIN.TXTRM-RF $DATA/Tmp.txtecho"Done :"

This script file uses Linux commands such as Rm,find, Cut, Sed,cat, and so on.

rm: Deleting files

find: Find Files

cut: Intercept path

sed: Add labels at the back of each line. In this example, the found *cat.jpg file is labeled 1 and the found *bike.jpg file is added with a callout of 2

Cat: Merges two categories into one file.

A train.txt file is eventually generated as follows:

Cat.jpg 1fish-bike.jpg 2

Of course, when there are few pictures, you can manually write this list file. But a lot of pictures, you need to use a script file to automatically generate. In future applications, the corresponding val.txt and test.txt files need to be generated in the same way.

Generated by this train.txt file, it can be used as a third parameter directly.

Next, let's take a look at the FLAGS parameter group and what it says:

-gray: Whether to open the picture as a grayscale image. The program calls the Imread () function in the OpenCV library to open the picture, which defaults to False

-shuffle: Random Shuffle of picture order. Default is False

-backend: Convert to DB file format, optional leveldb or Lmdb, default to Lmdb

-resize_width/resize_heighT: Change the size of the picture. In operation, all pictures are required to be of the same size, so the image size needs to be changed. The program calls the OpenCV Library's resize () function to zoom in on the image by default of 0, without changing

-check_size: Check that all data has the same size. The default is false, without checking

-encoded: Whether to put the original image encoding into the final data, the default is False

-encode_type: Corresponding to the previous parameter, encode the picture into which format: ' png ', ' jpg ' ...

Well, once we know these parameters, we can invoke the command to generate the final Lmdb format data.

Because of the more parameters, we can write a sh script to execute the command:

First, create the sh script file:

# sudo vi examples/images/create_lmdb.sh

Edit, enter the following code and save

#!/usr/bin/en shdata=examples/imagesrm-rf $DATA/img_train_lmdbbuild/tools/convert_imageset--  Shuffle--resize_height=256--resize_width=256 /home/xxx/caffe/examples/images/$DATA/train.txt $DATA/img_train _lmdb    

Set the parameter-shuffle to disturb the picture order. Set parameters-resize_height and-resize_width to change all picture sizes to 256*256.

/home/xxx/caffe/examples/images/the absolute path saved for the picture.

Finally, run the script file

# sudo sh examples/images/create_lmdb.sh

A folder named Img_train_lmdb is generated in the examples/images/directory, and the file is the db file we need.

Caffe Study Notes (10) Convert image data to db (Leveldb/lmdb) file