TensorFlow methods for reading and using different formats pretrained model in different training scenarios

analysis and examples of different application scenarios

TensorFlow read pre-training model is a common operation in model training, and the typical application scenarios include:

1) A restart is required after the training interruption, the previous checkpoint (including. data. Meta. Index checkpoint these four files) are saved, then the model is reloaded and the training or prediction continues from the last breakpoint. The implementation method is as follows: If the network structure diagram has been built in the code

With TF. Session () as Sess:
    Sess.run (Tf.global_variables_initializer ())
    saver = Tf.train.Saver ()
    ckpt = Tf.train.get_checkpoint_state (Os.path.dirname (' Checkpoints_path '))
    # If checkpoint exists then the training model before loading breakpoints if
    ckpt and Ckpt.model_checkpoint_path:
        saver.restore (Sess, Ckpt.model_checkpoint_path)

If you use someone else's complete checkpoint file, but you don't have the code to build the network structure, you can refactor the network through the. Meta file and check the weight data:

# ref:http://blog.csdn.net/spylyt/article/details/71601174 import TensorFlow as TF def Restore_from_meta (sess, Graph, Ckpt_path): With Graph.as_default (): Ckpt = Tf.train.get_checkpoint_state (ckpt_
            Path) if Ckpt and Ckpt.model_checkpoint_path:print (' Found checkpoint, try to restore ... ') Saver = Tf.train.import_meta_graph (". Join ([Ckpt.model_checkpoint_path, '. Meta '])) Saver.restore (Sess, CKPT.M
            Odel_checkpoint_path) # # # # # Print out the number of weights that can be trained in the network parameter name for VAR in tf.trainable_variables ():
        Print (VAR) # reads the desired variable weight according to the variable name, printing its shape conv1_2_weights = graph.get_tensor_by_name (' conv1_2/weights:0 ')  Print (conv1_2_weights.shape) if __name__ = = ' __main__ ': Ckpt_path = ' data\\voc2007_images\\gap_backup\\ ' graph = TF. Graph () Sess = tf. Session (graph=graph) Restore_from_meta (sess, graph, Ckpt_path) 

2) using the F.train.saver (). Restore (Sess, Ckpt_path) method to load the model will load all weights, if you want to load a specified number of weights (for example, when doing transfer learning), This can be done in the following ways:

# ref:http://blog.csdn.net/qq_25737169/article/details/78125061
sess = tf.  Session ()
var = tf.global_variables ()
Var_to_restore = [val for  Val in var if ' conv1 ' in Val.name or ' conv2 ' in Val.name]
saver = tf.train.Saver (var_to_restore)
Saver.restore (Sess, Os.path.join (Model_dir, Model_name))
Var_to_init = [val for  Val in var if ' conv1 ' not in Val.name or ' conv2 ' not in Val.name]
tf.initialize_variabl ES (Var_to_init)

In this way, the weights of CONV1 and conv2 are loaded, while the weights in the other layers are obtained through the network initialization.

If you use Tensorflowslim to build a network, the operation is much simpler:

exclude = [' layer1 ', ' layer2 ']
variables_to_restore = Slim.get_variables_to_restore (exclude=exclude)
Saver = Tf.train.Saver (Variables_to_restore)
Saver.restore (Sess, Os.path.join (Model_dir, Model_name))

This operation is a weight other than layer1 and layer2 for restore.

The above two situations are to build a network structure diagram or although you do not have a network structure diagram, but there is a corresponding. meta file on hand, you can refactor the network. If you only have the pre-trained model weights for the network, you want to use the weights of some layers in the model to initialize your network. The scene is different at this time.

The general model weights are saved as:

1) Checkpoint weights (Model_ckpt.data) or frozen graph (. pb) formatted file Model_ckpt.data weights are available through Tensorflow.python.pywrap_tensorflow. Newcheckpointreader or Tf.train.NewCheckpointReader obtain, the function of both, the following example:

Import OS from
Tensorflow.python import pywrap_tensorflow

Checkpoint_path = Os.path.join (Model_dir, "model_ Ckpt.ckpt ")
# Read data from checkpoint
reader = Pywrap_tensorflow. Newcheckpointreader (Checkpoint_path)
# reader = Tf.train.NewCheckpointReader (checkpoint_path) # Using the Newcheckpointreader method in Tf.train
Var_to_shape_map = Reader.get_variable_to_shape_map ()
# output weights tensor names and values
for key in Var_to_shape_map:
    print ("Tensor_name:", key)
    print (Reader.get_tensor (key))

If you want to initialize your own network with the pretrained weights of some layers, you can do it in sess by doing the following:

With Tf.variable_scope (", reuse = True):
        sess.run (Tf.get_variable (your_var_name). Assign (Reader.get_tensor ( Pretrained_var_name)))

The. PB weight graph can be obtained using the following method:

Import OS
import TensorFlow as tf

var1_name = ' pool_3/_reshape:0 '
var2_name = ' data/content:0 '

graph = Tf. Graph () with
Graph.as_default ():
    od_graph_def = tf. Graphdef ()
    with Tf.gfile.GFile (' MODEL_GRAPH.PB ', ' RB ') as FID:
        serialized_graph = Fid.read ()
        Od_graph_ Def. Parsefromstring (serialized_graph)
        var1_tensor, var2_tensor = Tf.import_graph_def (Od_graph_def, Return_ elements = [Var1_name, Var2_name])

Get the value of the variable 1 pool_3/_reshape:0 and the variable 2 data/content:0.

2) If the pre-training model is transferred by Caffe, the possible format is the NumPy file (. npy), which stores the weight data for each layer in a dictionary manner. Here's an example:

Import NumPy as NP
import cpickle

layer_name = "Conv1" with
open (' Model.npy ') as f:
    pretrained_weight = C Pickle.load (f)
    layer = Pretrained_weight[layer_name]
    weights = layer[0]
    biases = layer[1]

Get the variables in the CONV1, and then remove the weights and offsets separately, paying attention to the order of the weights.

If you want to initialize the CONV1 layer in your network with CONV1 weights and biases, you can use the following method:

With TF. Session () as Sess: With
    tf.variable_scope (Layer_name, reuse=true):
        for subkey, data in Zip (' weights ', ' biases ') , Pretrained_weight[layer_name]:
            sess.run (Tf.get_variable (subkey). Assign (data))

Summary

The above is a common pre-training model in TensorFlow operation, in the actual operation process according to their own scene needs to choose to use.