TensorFlow Advanced (update ...) __ Other Finishing---NLP

1. Configproto&gpu

Tf. Configproto is typically used when creating a session. Used to configure the session parameters

With TF. Session (config = tf. Configproto (...),...)
#tf. Configproto () parameter
log_device_placement=true: whether to print device allocation log
Allow_soft_placement=true: Allow TF to allocate devices automatically if the device you specify does not exist
TF. Configproto (Log_device_placement=true,allow_soft_placement=true)

Controlling utilization of GPU resources

Config = tf. Configproto ()
config.gpu_options.allow_growth = True session
= TF. Session (Config=config, ...)
# using Allow_growth option, just start allocating a small amount of GPU capacity, and then slowly increase on demand, because it will not release
#内存, so it will cause fragmentation

# Per_process_gpu_memory_fraction
GPU_OPTIONS=TF. Gpuoptions (per_process_gpu_memory_fraction=0.7)
CONFIG=TF. Configproto (gpu_options=gpu_options) session
= TF. Session (Config=config, ...)
#设置每个GPU应该拿出多少容量给进程使用, 0.4 on behalf of 40%

Control which GPU to use

~/cuda_visible_devices=0  python your.py# uses GPU0
~/cuda_visible_devices=0,1 python your.py# use gpu0,1
# Note that the word should not be wrong

#或者在 program opening
os.environ[' cuda_visible_devices '] = ' 0 ' #使用 GPU 0
os.environ[' cuda_visible_devices '] = ' 0,1 ' # using GPU 0,1

embedding_lookup ()

Use of Embedding_lookup ()
About the use of Embedding_lookup () in TensorFlow, in Udacity Word2vec will be involved, this article will be popular to explain.
Let's look at a simple demo:

#!/usr/bin/env/python
# coding=utf-8
import tensorflow as tf
import numpy as np

input_ids = Tf.placeholder (Dtype=tf.int32, Shape=[none])

embedding = tf. Variable (Np.identity (5, Dtype=np.int32))
input_embedding = Tf.nn.embedding_lookup (embedding, Input_ids)

Sess = tf. InteractiveSession () Sess.run (Tf.global_variables_initializer ()) print (
embedding.eval ())
print ( Sess.run (Input_embedding, Feed_dict={input_ids:[1, 2, 3, 0, 3, 2, 1)})

The code first uses Palceholder to define an unknown variable input_ids used to store the index, and a known variable embedding is a 5*5 diagonal matrix.
The results of the operation are:

embedding = [[1 0 0 0 0] [0 1 0 0 0] [0 0 1 0 0] [0 0 0 1] [0 0 0 0 0
             ]] the
input_embedding = [[1] 0 1 0 0]
                   [0 0 1 0 0] [0 0 0 1 0] [1 0 0 0 0] [0 0 0 1 0] [0 0 1 0 0] [0 1 0 0]
                   ]

The simple thing is to look for the corresponding element in the embedding according to the ID in the input_ids. For example, input_ids=[1,3,5], then find the embedding subscript 1,3,5 vector to form a matrix return.

If you rewrite input_ids to the following format:

input_embedding = Tf.nn.embedding_lookup (embedding, Input_ids)
print (Sess.run (input_embedding, feed_dict={ Input_ids:[[1, 2], [2, 1], [3, 3]]})

The output turns into the following format:

[[[0 1 0 0 0] [0 0 1 0 0]] [[0 0 1 0] [0 0 1 0 0]
 ] [
 [
  0] 0 0 0 1] [0 0 0 0 1]]]

Comparison of two results is not difficult to find, equivalent to the Np.array in the direct use of subscript array to obtain data. The detail to be noted is that the returned tensor dtype is consistent with the dtype of the incoming queried tensor, regardless of the dtype of IDs. Tf.nn.dropout

Prevent fitting
Tf.nn.dropout is a function that is used in tensorflow to prevent or mitigate the fitting, and is generally used in a fully connected layer.

Dropout is the random throw away of a subset of neurons during different training processes. That is, the activation value of a neuron with a certain probability p, let it stop working, the training process does not update the weight, nor participate in the calculation of the neural network. But its weight has to be preserved (only temporarily not updated), because it may have to work again the next time the sample is entered. The schematic diagram is as follows:

Tf.nn.dropout (
    x,  # A floating point tensor.
    Keep_prob,   #A scalar Tensor with the same type as x. The probability, each element is kept.
    Noise_shape=none,  #A 1-d Tensor of type Int32, representing the shape for randomly generated flags.
    Seed=none,   #A Python Integer. Used to create random seeds. Tf.set_random_seed for behavior.
    Name=none  #A name for this operation (optional).
)