Enable the data set generated by Caffe to run directly on Theano (1)-lmdb, protobuf, and caffetheano

Enable the data set generated by Caffe to run directly on Theano (1)-lmdb, protobuf, and caffetheano

No matter which framework is used for CNNs training, there are three types of datasets:

The Training Set is used to train the network.

The Validation Set is used to test the network accuracy during training.

Test Set is used to Test the final accuracy after network training is completed.

 

Caffe generates data in two formats: Lmdb and Leveldb.

They are all Key/Value Pair Embedded Database Management System Programming libraries.

Although the memory consumption of lmdb is 1.1 times that of leveldb, the speed of lmdb is 10% to 15% faster than that of leveldb. More importantly, lmdb allows multiple training models to read the same set of datasets at the same time.

Therefore, lmdb replaces leveldb as the default dataset generation format for Caffe.

 

Google Protocol Buffer Installation

Protocol Buffer is an automatic mechanism similar to XML for serializing data.

First download the latest version in Protocol Buffers:

Https://developers.google.com/protocol-buffers/docs/downloads

Decompress the package and run:

./configure$ make$ make check$ make installpip installprotobuf

Add dynamic link library

exportLD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

 

Lmdb Installation

pip install lmdb

To parse (parse) A protobuf-type data, first tell the computer what format the protobuf data is in (which items are available, the data type of each of these items determines how many bytes are occupied. Can these items be repeated ), after protobuf is installed, the module can use the special Syntax of protobuf to define these formats (this is. proto file), and then use protoc to compile this. the proto file can generate the target file you need.

To define your own. proto file, please read:

Https://developers.google.com/protocol-buffers/docs/proto? Hl = zh-cn

 

Compile the. proto File

protoc--proto_path=IMPORT_PATH --cpp_out=DST_DIR --java_out=DST_DIR--python_out=DST_DIR path/to/file.proto

-- Proto_path can also be abbreviated as-I is the path where. proto is located

Output path:

-- Cpp_out: generate the available header files for C ++, which are ***. pb. h (including declarative class )***. pb. cc (including executable classes). When using this function, you only need to include "***. pb. h"

-- Java_out: Generate available java header files

-- Python_out: generate the available python header file ** _ pb2.py. import ** _ pb2.py when using it.

The last parameter is the complete path of your. proto file.

Welcome to the discussion and follow up on this blog, Weibo, and zhihu personal homepage for further updates ~

Reprinted, please respect the work of the author and keep the above text and link of the article completely. Thank you for your support!