Use of HDF5 format files on Caffe and regression (regression) issues

Original:

Http://www.cnblogs.com/hrlnw/p/5824780.html

Recently used Caffe to do a regression problem, first with the data layer in the Data,float_data tried, data used to store pictures, Float_data store regression storage label, Found not, because Float_data and label are pointing to the same block of memory, the two can not coexist, unless the change Caffe source code. I took a closer look at the source of Caffe, I found that this place is not right, specifically, Caffe read image and label is such a process:

1. Write program converts image and label into datum format and writes to Lmdb

The 2.Data layer reads the datum format data from the Lmdb and converts it into a batch

3. Copy the image and label data from the batch to top memory or RAM

The above 2, 3 steps can be viewed Data_layer.cpp,base_data_layer.cpp and base_data_layer.cu three files, which have nothing to do with float_data, so even if you put regression Values are stored in Datum's float_data, and Data layer does not read it. I currently think of the only two ways, or to change the Caffe source code, or Lmdb and hdf5 together, the source code is afraid to make mistakes, I tried the HDF5 format to do input, although it was done, but there are pits.

To say how to write data to the Hdf5 file, I use the Python,numpy format array can be stored directly, the following methods:

#1. First create an empty numpy array
hdf_data = Np.zeros (len (all_samples), num), Dtype=np.float32)

#2. Fill Hdf_data

# 3. Open the H5 file and write to Hdf_data
h5_file = h5py. File (H5_path, ' W ')
h5_file.create_dataset (' Hdf_data ', data=hdf_data)
h5_file.close ()
# Write the H5 file path to a txt
h5txt_file = open (H5txt_path, ' W ')
h5txt_file.write (H5_path)
h5txt_file.close ()

In the Prototxt file that defines the network, this uses the

Layer {
   name: "Hdf5_data"
   type: "Hdf5data" top
   : "Hdf_data"
   include {
     phase:train
   }
   Hdf5_data_param {
     Source: ' Your_h5txt_file_path '
     batch_size:1024
   }
 }

Where source is the only one line of txt path. Top is the name of the block of data that you need to use in the H5.

It should be noted that, if the image data, preprocessing to do first, and then deposit into the H5 file, because the Hdf5data layer does not support the preprocessing function. In addition there is a pit, as far as possible do not put the picture in the H5 file, because the Hdf5data layer can not follow the batch to read data, only one time to all the data from the H5 file read into memory, if the H5 file is very large, will be the error, specifically hanging in the Hdf5data implementation code , there is an assertion of the size of the data. So you don't want to change the Caffe code and want to ensure that the training program can run the following methods:

Put regression values in accordance with the HDF5 format, image and label according to normal data stored in Lmdb can. You can read data from different sources in the prototxt that defines the network, such as the following method:

Layer {
  name: ' Lmdb_data '
  type: ' Data ' top
  : ' Data ' top
  : ' label '
  include {
    phase:train
  }
  Data_param {
    Source: ' Train_data '
    batch_size:1024
    backend:lmdb
  }
  Transform_param {
  	mean_value:128
  	scale:0.0078125
  }
}
layer {
  name: ' Hdf5_data '
  type: ' Hdf5data ' Top
  : ' bbox '
  include {
    phase:train
  }
  hdf5_data_param {
    Source: ' Hdf5_train.txt '
    batch_size:1024
  }
}