Let caffe generate data set to run directly on Theano (a)--lmdb and protobuf

Regardless of the framework used for CNNs training, there are 3 data sets:

The Training set is used to train the network.

Validation set is used to test network accuracy during training.

Test set is used for testing the final correct rate after network training is completed.

The data generated by Caffe is divided into 2 formats: Lmdb and LEVELDB.

They are key/value pairs (key/value pair) embedded database management System programming library.

While Lmdb memory consumption is 1.1 times times leveldb, Lmdb is 10% to 15% faster than LEVELDB, and more importantly LMDB allows multiple training models to read the same set of data simultaneously.

As a result, Lmdb replaces leveldb as the default dataset generation format for Caffe.

Installation of Google Protocol buffer

Protocol buffer is an XML-like automatic mechanism for serializing data.

First download the latest version in Protocol buffers:

Https://developers.google.com/protocol-buffers/docs/downloads

After decompression run:

./configure$ make$ make check$ make Installpip installprotobuf

Add a dynamic link library

Exportld_library_path=/usr/local/lib: $LD _library_path

Installation of Lmdb

Pip Install Lmdb

To parse (parse) a protobuf type of data, first tell the computer what format is inside of the PROTOBUF data (what are the items, what data types are used to determine how many bytes are occupied, whether the items can be duplicated, repeated several times), Install PROTOBUF This module can use the PROTOBUF syntax to define these formats (this is the. proto file), and then use the PROTOC to compile the. proto file to generate the target file you need.

To define your own. proto file, read:

Https://developers.google.com/protocol-buffers/docs/proto?hl=zh-cn

Compiling the. Proto file

Protoc--proto_path=import_path--cpp_out=dst_dir--java_out=dst_dir--python_out=dst_dir Path/to/file.proto

--proto_path can also be abbreviated to-I is the path where the. Proto is located

Output path:

--cpp_out to generate the header files that are available for C + +, ***.pb.h (including the Declaration Class) ***.pb.cc (including executable classes), when used as long as include "***.pb.h"

--java_out generating a Java-usable header file

--python_out generate Python-Usable header files, **_pb2.py, import**_pb2.py when used

The last parameter is your. proto file full path.

Welcome to participate in the discussion and follow This blog Weibo personal home Subsequent content continues to update Oh ~

reproduced please respect the author's labor, complete reservations The above text as well article links , thank you for your support!

Let caffe generate data set to run directly on Theano (a)--lmdb and protobuf