Caffe Code Guide (4): Data Set preparation

Caffe Code Guide (4): Data Set preparation

Caffe There are two simple examples: Mnist and CIFAR-10, the former is used for handwritten numeral recognition, the latter for small image classification. These two datasets can be downloaded in the CAFFE source framework using scripts (caffe_root/data/mnist/get_mnist.sh and caffe_root/data/cifar10/get_cifar10.sh), as shown in:

[Plain]View Plaincopyprint?

2. $./get_cifar10.sh
4. Downloading ...
6. --2014-12-02 01:20:12--http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz
8. Resolving www.cs.toronto.edu ... 128.100.3.30
10. Connecting to www.cs.toronto.edu|128.100.3.30|:80 ... Connected.
12. HTTP request sent, awaiting response ... OK
14. length:170052171 (162M) [Application/x-gzip]
16. Saving to: "Cifar-10-binary.tar.gz"
22. 100%[========================================================================================================== =================================================>] 170,052,171 859k/s in 2m 16s
28. 2014-12-02 01:22:28 (1.20 mb/s)-"cifar-10-binary.tar.gz" saved [170052171/170052171]
34. Unzipping ...
36. Done.
38. $ ls
40. Batches.meta.txt data_batch_1.bin data_batch_2.bin data_batch_3.bin data_batch_4.bin data_batch_5.bin get_cifar10.sh Readme.html Test_batch.bin

[Plain]View Plaincopyprint?

2. $./get_mnist.sh
4. Downloading ...
6. --2014-12-02 01:24:25--http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
8. Resolving yann.lecun.com ... 128.122.47.89
10. Connecting to yann.lecun.com|128.122.47.89|:80 ... Connected.
12. HTTP request sent, awaiting response ... OK
14. length:9912422 (9.5M) [Application/x-gzip]
16. Saving to: "Train-images-idx3-ubyte.gz"
22. 100%[========================================================================================================== =================================================>] 9,912,422 2.09m/s in 6.7s
28. 2014-12-02 01:24:33 (1.42 MB/s)-"train-images-idx3-ubyte.gz" saved [9912422/9912422]
34. --2014-12-02 01:24:33--http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
36. Resolving yann.lecun.com ... 128.122.47.89
38. Connecting to yann.lecun.com|128.122.47.89|:80 ... Connected.
40. HTTP request sent, awaiting response ... OK
42. length:28881 (28K) [Application/x-gzip]
44. Saving to: "Train-labels-idx1-ubyte.gz"
50. 100%[========================================================================================================== =================================================>] 28,881 42.0k/s in 0.7s
56. 2014-12-02 01:24:34 (42.0 kb/s)-"train-labels-idx1-ubyte.gz" saved [28881/28881]
62. --2014-12-02 01:24:34--http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
64. Resolving yann.lecun.com ... 128.122.47.89
66. Connecting to yann.lecun.com|128.122.47.89|:80 ... Connected.
68. HTTP request sent, awaiting response ... OK
70. length:1648877 (1.6M) [Application/x-gzip]
72. Saving to: "T10k-images-idx3-ubyte.gz"
78. 100%[========================================================================================================== =================================================>] 1,648,877 552k/s in 2.9s
84. 2014-12-02 01:24:39 (552 kb/s)-"t10k-images-idx3-ubyte.gz" saved [1648877/1648877]
90. --2014-12-02 01:24:39--http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
92. Resolving yann.lecun.com ... 128.122.47.89
94. Connecting to yann.lecun.com|128.122.47.89|:80 ... Connected.
96. HTTP request sent, awaiting response ... OK
98. length:4542 (4.4K) [Application/x-gzip]
100. Saving to: "T10k-labels-idx1-ubyte.gz"
106. 100%[========================================================================================================== =================================================>] 4,542 19.8k/s in 0.2s
112. 2014-12-02 01:24:40 (19.8 kb/s)-"t10k-labels-idx1-ubyte.gz" saved [4542/4542]
118. Unzipping ...
120. Done.
122. $ ls
124. get_mnist.sh t10k-images-idx3-ubyte t10k-labels-idx1-ubyte train-images-idx3-ubyte train-labels-idx1-ubyte

If you download a problem can be obtained from my resources, url http://download.csdn.net/detail/kkk584520/8213463.

The original dataset is a binary file and needs to be converted to LEVELDB or Lmdb to be Caffe recognized. The conversion Format tool is already integrated in the Caffe code, see Caffe_root/examples/mnist/convert_mnist_data.cpp

And Caffe_root/examples/cifar10/convert_cifar_data.cpp, if you are unfamiliar with leveldb or lmdb operations, you can learn from both source codes. We only need to execute two commands in the Caffe_root directory:

./examples/mnist/create_mnist.sh

./examples/cifar10/create_cifar10.sh

Caffe Code Guide (4): Data Set preparation