Deep learning Caffe platform--making your own. Lmdb format DataSet and classification label file

Caffe for training data formats, support: Lmdb, h5py ..., where lmdb data formats are commonly used for single-label data, such as classification, often using LMDB data formats. For problems such as regression, or multi-label data, the format of the h5py data is generally used. Of course, it seems that there are other formats of data available, this article mainly for the Lmdb data format of the production method, a brief explanation.
Take a simple example of how to generate your own set of images. It is divided into two main steps:
(1) Generate a list of label files, i.e. generate a. txt file (2) Connect the image in the TXT file list with the image of the train image Library, the Val Image Library, and generate the Lmdb format file
Lmdb Data

Lmdb is used for single-label data. For the sake of simplicity, I followed a gender classification as an example to make an explanation of the relevant data.

1. Data Preparation

First we have to prepare the training data and then create a new folder named Train and a Val folder:

The train file holds the training data, and the Val file holds the validation data. Then we are under the train file, the training data cat, dog pictures are placed under a folder:

Similarly, we also create folders below the Val file:

Two files are also stored separately we use to verify the picture data of cat and dog image files. We stored pictures of cats under the Test_cat, and then stored the dog pictures of the data under Test_dog.

Place the train and Val folders in a folder, and this article is placed in the Data_test folder.

2, label file. txt file production.

Then we need to make a train.txt, Val.txt file, these two files contain the image path of the training data above, and its corresponding tag, as shown below.

We label the cat picture as 1, and the dog picture is labeled 0. The generation of TXT in the label data file can be obtained by scanning the path of the cat, dog, and Train.txt and val.txt by the following code:

Note that there is a space between the path and the label , and the label is best to start at 0.

[Python]  View plain  copy   <pre class= "python"  name= "code" > #coding:utf-8   ""   created on jul 29, 2016    @author: sgg  '        "<span style=" "font-family:arial;font-size:18px;" " > "  " <span style= "" font-size:18px; "" > "  " <span style= "" font-size:18px; "" >     import os      def issubstring (SUBSTRLIST,STR):       flag=True       for substr in substrlist:            if not (SUBSTR&NBSP;IN&NBSP;STR):               flag=False               return flag      #扫面文件    def  getfilelist (findpath,flagstr=[]):        FileList=[]       filenames=os.listdir ( Findpath)        if len (FileNames) >0:            for fn in FileNames:                if len (FLAGSTR) >0:                    if issubstring (FLAGSTR,FN):                         fullfilename=os.path.join (FINDPATH,FN)                         filelist.append (Fullfilename)                else:     &nbs