Variables: Create, Initialize, save, and load

Introduction

When you train a model, you use variables to save and update parameters. The variable in TensorFlow is the amount of tensor (tensor) that is saved in the memory buffer. They must be displayed for initialization and can be saved to disk after training is completed. After that, you can reload these values for testing and model analysis.
This document refers to the following TensorFlow class. The following links point to their more detailed API:TF. Variable class. Tf.train.Saver class. Create

When you create a variable, you pass a tensor data as its initial value to the variable () constructor. TensorFlow provides a bunch of operations that generate tensor data from constants or random values for initialization.
Note These operations require you to specify the shape of the tensor data. This shape automatically becomes the shape of the variable. The shape of the variable is fixed. However, TensorFlow provides some advanced mechanisms for changing the shape of a variable.

# Create two variables
weights = tf. Variable (Tf.random_normal ([784, $], stddev=0.35), name= "weights")
biases = tf. Variable (Tf.zeros ([+]), name= "biases")

Call TF. Variable () adds these nodes to the calculation diagram: A variable node that holds the value of the variable an initialization action node that sets the variable to the initial value. It is actually a tf.assign node. The initial value node, such as the zeros node in the example, is also added to the calculation diagram.

Tf. The Variable () return value is an instance of the Python class tf.variable. Device Configuration

A variable can be plugged into a developed device when it is created by using the WITH Tf.device (...)::

# plug the variable into the CPU with
tf.device ("/cpu:0"):
  v = tf. Variable (...)

# plug the variable into the GPU with
tf.device ("/gpu:0"):
  v = tf. Variable (...)

# plug the variable into the specified parameter service task with
tf.device ("/job:ps/task:7"):
  v = tf. Variable (...)

note that some operations that change variables, such as v.assign () and the update operation of variables in Tf.train.Optimizer, must be run on the same device as when they were created with the variable. The creation of these operations is that incompatible device configurations will be ignored. Initialize

The initialization of a variable must be found before other operations of the model, and must be shown to run. The simplest way is to add a node to initialize all the variables, and then run the node before using the model.
Or you can choose to load the variable from the checkpoint file, which will be described later.
Use Tf.global_variables_initializer () to add a node to initialize all variables. After you have built the complete model and loaded the model in the session, run the node.

# Create two variables
weights = tf. Variable (Tf.random_normal ([784, $], stddev=0.35),
                      name= "weights")
biases = tf. Variable (Tf.zeros ([+]), name= "biases") ...
# Add the node used to initialize the variable
init_op = Tf.global_variables_initializer ()

# Then, when loading the model with
TF. Session () as Sess:
  # Run initialization operation
  Sess.run (init_op) ...
  # Use Model
  ...

initialize from another variable

Sometimes you need to use another variable to initialize the current variable. Since Tf.global_variables_initializer () adds nodes that are suitable for parallel initialization of all variables, you have to be cautious if you have this requirement.
To initialize a new variable from another variable, use another method of the variable Initialized_value (). You can use the initial value of the old variable directly as the initial value of the new variable, or you can perform some operations on the original value of the old variable and then use it as the initial value of the new variable.

# Create a variable with a random number
weights = tf. Variable (Tf.random_normal ([784, Max], stddev=0.35),
                      name= "weights")
# Create another variable that has the same initial value as weights
= TF . Variable (Weights.initialized_value (), name= "W2")
# creates another variable whose initial value is weights twice times
w_twice = tf. Variable (Weights.initialized_value () * 2.0, Name= "W_twice")

Custom Initialization

Tf.global_variables_initializer () is very convenient to initialize all the variables one step. You can also pass the specified list to it, initializing only the variables in the list. For more options, see variables documentation, including checking that variables are initialized. Save and load

The simplest way to save and load a model is to use the Tf.train.Saver object. Its constructor adds the save and restore nodes on the calculation diagram, targeting all or specified variables on the graph. The Saver object provides a way to run these nodes, as long as you specify a file for read-write checkpoint. checkpoint file

Variables are stored in a binary file in the checkpoint file, which is roughly a mapping of the variable name to the tensor value
When you create a Saver object, you can select the variable name in the checkpoint file. By default, it uses Variable.name as the name of the variable.
To understand what variables are in the checkpoint file, you can use the Inspect_checkpoint library and use the Print_tensors_in_checkpoint_file function in more detail. Save Variables

Use Tf.train.Saver () to create a Saver object, and then use it to manage all the variables in the model.

# Create some variables
v1 = tf. Variable (..., name= "v1")
v2 = tf. Variable (..., name= "v2") ...
# Add the node used to initialize the variable
init_op = Tf.global_variables_initializer ()

# Add a node for saving and loading all variables
saver = Tf.train.Saver ()

# Then, load the model, initialize all the variables, and after doing something, save the variable to disk with
TF. Session () as Sess:
  sess.run (INIT_OP)
  # do some things.
  # Save the variable to disk
  Save_path = Saver.save (Sess, "/tmp/model.ckpt")
  print ("Model saved in file:%s"% Save_path)

Load Variables

The Saver object can also be used to load variables. Note that when you load variables from a file, you do not have to implement them to initialize them.

# Create two variables
v1 = tf. Variable (..., name= "v1")
v2 = tf. Variable (..., name= "v2") ...
# Add a node for saving and loading all variables
saver = tf.train.Saver ()

# Then, load the model, use the Saver object to load the variable from the disk, and then use the model for some operations with
TF. Session () as Sess:
  # Load object from disk
  Saver.restore (sess, "/tmp/model.ckpt")
  print ("model restored.")
  # Use the model for some action ...
  

Select variables to save and load

If you do not pass any parameters to Tf.train.Saver (), the Saver object will process all the variables in the diagram. Each variable is saved on disk using the name passed to it when it was created.
Sometimes, we need to display the name of the variable stored in the checkpoint file. For example, you might use a variable called "weights" to train the model, and you want to save it with a name of "params".
Sometimes, we only save and load part of the parameters of the model. For example, you have trained a 5-layer neural network, and now you want to train a new neural network that has 6 layers. Load the parameters of the old model as the first 5 layers of the new neural network.
By passing it to Tf.train.Saver () a Python dictionary, you can simply specify the name and the variable you want to save. The keys of the dictionary are the names that are stored on the disk, and values is the value of the variable.
Note:
If you need to save and load variables of different subsets, you can create as many saver objects as you want. The same variable can be saved by multiple Saver objects. Its value is changed only after the restore () method has run.
If you load only a subset of variables at the beginning of the session, you also need to run initialization for the other variables. See Tf.initialize_variables () for more information.

# create some objects v1 = tf. Variable (..., name= "v1") v2 = tf. Variable (..., name= "v2") ... # Add a node for saving and loading variables v2, using the name "MY_V2" saver = Tf.train.Saver ({"MY_V2": V2}) # Use the Saver object
Normally after. ...