Deep learning to practice, an indispensable path is to the intelligent terminal, embedded equipment and other directions. But the terminal device does not have the powerful performance of GPU server, how to make the end device application deep learning?
Fortunately, Google has launched the tfmobile, last year further, the introduction of Tflite, its application of the idea of using migration learning to train their own model on the GPU server, and then porting the customized model to Tflite,
The terminal equipment only uses the model to make forward inference and predicts the result. This article is based on the following three articles:
- Theory article: Www.tensorflow.org/hub/tutorials/image_retraining#other_architectures
- Practice Article One: codelabs.developers.google.com/codelabs/tensorflow-for-poets/index.html#0
- Practice Chapter Two: codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-tflite/#0
- Google's pre-compiled model: GITHUB.COM/TENSORFLOW/TENSORFLOW/BLOB/MASTER/TENSORFLOW/CONTRIB/LITE/G3DOC/MODELS.MD
- Toco Official website: Github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/lite/toco
You can also easily customize your image recognition application after you have mastered it.
The first step. Preparing data
Data are: http://download.tensorflow.org/example_images/flower_photos.tgz
This is a collection of pictures about the flower classification, after downloading the decompression, you can see that there are 5 varieties of categories: Daisy (Daisy), Dandelion (dandelion), Rose (Rose), sunflower (sunflower), Tulip (tulip).
Our goal is to get a model of the pattern recognition by re-training the precompiled model.
The second step. Re-training
1. Pick Precompiled model
From the "pre-compiled model provided by Google" list, we can generally see that there are two types of models, one is the float Models (floating point model), one is the quantized Models (quantitative model), what is the difference?
In fact, the float models is represented as a high-precision model, which means that the model size is larger, the recognition accuracy is higher, the recognition time is longer, is suitable for high-performance terminal equipment, and the quantized models is the low-precision model, whose precision takes a fixed 8-bit size, Therefore, the model size is small, the recognition accuracy is low, the recognition time is short, suitable for low performance terminal equipment, more detailed description can be see www.tensorflow.org/performance/quantization.
Our mobile devices are rapidly upgrading and can generally be used with the float Models. In this model, there are a lot of pre-compiled models to choose, for this article, the main focus is inception and mobilenet two architectures.
Note Mobilenet is actually divided into many kinds, such as mobilenet_v1_0.50_224, where the third parameter is the model size scale value (only approximate, not accurate), divided into 0.25/0.50/0.75/1.0 four scale values, the fourth parameter is the picture size, It has a value of 128/160/192/224 four values.
Interested in looking at each model hierarchy can be viewed in the following code:
import tensorflow as tf
import tensorflow.gfile as gfile
MODEL_PATH = '/home/yourname/Documents/mobilenet_v1_1.0_224/frozen_graph.pb'
with tf.Graph().as_default() as graph:
with gfile.FastGFile(MODEL_PATH, 'rb') as f:
graph_def = tf.GraphDef()
for op in graph.get_operations():
for tensor in op.values():
if __name__ == '__main__':
Considering the performance of the test phone is not bad, we chose mobilenet_v1_1.0_224 this version as our pre-compiled model.
2. Download the Training Code
You need to download the training model code and the Android-related code as follows:
git clone https://github.com/googlecodelabs/tensorflow-for-poets-2
Among them, the scripts directory of retrain.py is our concern, this code currently only supports INCEPTION_V3 and mobilenet two pre-compilation models, the default training model is INCEPTION_V3.
3. Retrain the Model
The training commands of the two models are different, if you go to the default Inception_v3 model, you can use the following command:
python -m scripts.retrain \
If you go to the Mobilenet model, you can use the following command:
python -m scripts.retrain \
The model commands are interpreted as follows:
--architecture is a schema type that supports Mobilenet and inception_v3 two types of
4. Test the effectiveness of training
--image_dir is the data address, assuming you set up the Tflite directory under the Tensorflow-for-poets-2 directory, put the flower picture set into it
--output_labels the last training to generate the label of the model, because the flower picture collection has been categorized according to subdirectories, so Retrained_labels.txt finally contains the above five categories of flowers category name
--output_graph the last training-generated model
After the--model_dir command is started, the pre-compilation model's
--how_many_training_steps the number of training steps, default is 4000 if not specified
--bottleneck_dir is used to cache the top layer's training data as a file.
--learning_rate Learning Rate
In addition, there are some parameters that can be adjusted as needed:
--testing_percentage divide the picture by how much as the test data, the default is 10
--validation_percentage divide the picture by how much as the validation data, the default is 10, when the two values are set, the ratio of training data to 80%
--eval_step_interval How many steps to perform an evaluation, the default is 10
--train_batch_size number of pictures in a single session, default is 100
--validation_batch_size number of images validated at once, default is 100
--random_scale given a scale value, then randomly expands the size of the training picture by default to 0
--random_brightness given a scale value, then randomly increases or decreases the brightness of the training picture by default to 0
--random_crop given a scale value, then randomly cropping the edge value of the training picture, which defaults to 0
We use mobilenet_1.0_224 to train, and then find a picture to see if we can recognize it correctly:
python -m scripts.label_image \
The result is:
Evaluation time (1-image): 1.010s
or accurately identify the daisy out of the way.
5. Convert model format
PB format is not able to run on the Tflite, Tflite absorbed Google's Protobuffer advantages, created the Flatbuffer format, the specific performance is the suffix. tflite file.
The above Toco's official website has described how to convert the PB format to a tflite file via the command line, or the format can be converted in code. Not only supports the PB format, but also supports the conversion of the HDF5 file format into Tflite, which enables the sharing of models with other frameworks.
How does that turn out? This example is converted by command line mode. If the training model is INCEPTION_V3, the command line is as follows:
If the training model is mobilenet, the command line mode is as follows:
Some points need to be explained:
The--input_array parameter represents the portal tensor op name for the model diagram structure, and the entry name for the mobilenet is input,inception_v3, the entry name is Mul, why? You can view the contents of the scripts/retrain.py code:
if architecture == 'inception_v3':
# pylint: disable=line-too-long
data_url = 'http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz'
# pylint: enable=line-too-long
bottleneck_tensor_name = 'pool_3/_reshape:0'
bottleneck_tensor_size = 2048
input_width = 299
input_height = 299
input_depth = 3
resized_input_tensor_name = 'Mul:0'
model_file_name = 'classify_image_graph_def.pb'
input_mean = 128
input_std = 128
data_url = 'http://download.tensorflow.org/models/mobilenet_v1_'
data_url += version_string + '_' + size_string + '_frozen.tgz'
bottleneck_tensor_name = 'MobilenetV1/Predictions/Reshape:0'
bottleneck_tensor_size = 1001
input_width = int(size_string)
input_height = int(size_string)
input_depth = 3
resized_input_tensor_name = 'input:0'
The Resized_input_tensor_name is the entry name for the newly generated model, and you can view the newly generated model hierarchy by using the code visualization "1. Pick precompiled model" above. So the name must be correctly written, otherwise running the command throws the exception "Valueerror:invalid tensors ' input ' were found".
The--output_array is the export name of the model. Why is Final_result this name, because in scripts/retrain.py there are:
The name of the output classification layer in the retrained graph.\
That is, the export name defaults to Final_result.
--input_shape Note that the Mobilenet training picture size is 224, while the training image size of Inception_v3 is 299.
Finally Optimized_graph.lite is the model file that we will be porting to Android.
The third step. Android Tflite
1. Download Android Studio
This step is not the focus of this article, please download and install the latest SDK and NDK on your own developer.android.com/studio/.
2. Introduction of Engineering
Introducing Tensorflow-for-poets-2/android/tflite code from Android Studio, there are four classes, three classes dealing with layouts, and we just need to focus on the Imageclassifier.java class.
3. Importing models
The generated model can be imported into the resource directory of the above project through the command line method:
4. Modify the Imageclassifier.java class
cp tf_files/optimized_graph.lite android/tflite/app/src/main/assets/mobilenet.lite
cp tf_files/retrained_labels.txt android/tflite/app/src/main/assets/mobilenet.txt
Note that you can modify four places:
5. Run the viewing effect
/** Name of the model file stored in Assets. */
Private static final String MODEL_PATH = "mobilenet.lite";
/** Name of the label file stored in Assets. */
Private static final String LABEL_PATH = "mobilenet.txt";
Static final int DIM_IMG_SIZE_X = 224; //If inception, change to 299
Static final int DIM_IMG_SIZE_Y = 224; // If inception, change to 299
Once you've connected your phone, clicking "Run" and "run" will deploy the app to your phone, and any images captured by the camera will be ranked according to the 5 categories in the label.
We can use Baidu search some of these five categories of flowers to identify, to see the correct rate of recognition.
PostScript: According to my test results, in the picture collection of flowers, the new model recognition rate of mobilenet_1.0_244 model is higher, and the recognition rate of the new model generated by the INCEPTION_V3 model is low or not allowed.
It is recommended that the new data set be compared between the two models to find the model that best suits you.