Keras Depth Training 4:gpu settings

4.1 Keras specifying runtime graphics and limiting GPU usage

https://blog.csdn.net/A632189007/article/details/77978058

#!/usr/bin/env python
# encoding:utf-8

"" "
@version: python3.6
@author: Xiangguo Sun
@contact: sunxiangguo@seu.edu.cn
@site: http://blog.csdn.net/github_36326955
@software: Pycharm
@file: 2clstm.py
@time: 17-7-27 5:15pm "" "

import os
import TensorFlow as TF
import Keras.backend.tensorflow_backend as KTF

#进行配置, each GPU uses 60% upper limit existing
os.environ["Cuda_visible_devices"]= "#" Use GPU Config = tf with number 1 and number 2nd
. Configproto ()
config.gpu_options.per_process_gpu_memory_fraction = 0.6 # Each GPU has an existing last control within 60%
session = TF. Session (Config=config)

# Set Session
Ktf.set_session (session)

4.2 using Fit_generator to minimize memory occupancy ratio/Data batch

#!/usr/bin/env python # encoding:utf-8 "" "@version: python3.6 @author: Xiangguo Sun @contact: sunxiangguo@seu.edu.cn @s ite:http://blog.csdn.net/github_36326955 @software: Pycharm @file: 2clstm.py @time: 17-7-27 5:15pm "" "#将内存中的数据分批 (Batch _size) sent to memory for Operation Def generate_arrays_from_memory (Data_train, Labels_cat_train, batch_size): x = Data_train Y=labels
        _cat_train Ylen=len (y) loopcount=ylen//batch_size while true:i = Np.random.randint (0,loopcount)
Yield x[i*batch_size: (i+1) *batch_size],y[i*batch_size: (i+1) *batch_size] # The following load will not occupy the display, but load into memory. Data_train=np.loadtxt ("./data_compress/data_train.txt") labels_cat_train=np.loadtxt ('./data_compress/labels_cat _train.txt ') data_val=np.loadtxt ('./data_compress/data_val.txt ') labels_cat_val=np.loadtxt ('./data_compress/
                                                   Labels_cat_val.txt ') hist=model.fit_generator (Generate_arrays_from_memory (Data_train,
       Labels_cat_train,                                            BS), Steps_per_epoch=int (Train_size/bs), Epochs=ne, Validation_data= (data_val,labels_cat_val), CALLB Acks=callbacks)

If your memory is not enough, you can also choose to import batches from the file:

#!/usr/bin/env Python # encoding:utf-8 "" "@version: python3.6 @author: Xiangguo Sun @contact: sunxiangguo@seu.edu.cn @site: http://blog.csdn.net/github_36326955 @software: pycharm @file: 2clstm.py @time: 17-7-27   
    5:15pm "" "Def Process_line (line): TMP = [Int (val) to Val in Line.strip (). Split (', ')] x = Np.array (Tmp[:-1])  
        y = Np.array (tmp[-1:]) return x, y def generate_arrays_from_file (path,batch_size): While 1: f = open (path) CNT = 0 X =[] Y =[] for line in F: # Create Numpy ar  
            rays of input data # and labels, from all line in the file x, y = process_line (line)  X.append (x) y.append (Y) cnt + = 1 if cnt==batch_size:cnt  
= 0 Yield (np.array (x), Np.array (Y)) X = [] Y = [] F.close () 

More references
https://blog.csdn.net/sinat_26917383/article/details/74922230
https://blog.csdn.net/lujiandong1/article/details/54869170 4.3 nvidia-smi command to view GPU status

Nvidia-smi-l

The above command allows for real-time tracking
command interpretation from the blog: https://blog.csdn.net/sallyxyl1993/article/details/62220424

This is the information for Tesla K80 on the server.
In the table above:
The first column of the fan:n/a is the fan speed, from the 0到100% change between, this speed is the computer expected fan speed, in fact, if the fan blocked, may not hit the display speed. Some devices do not return rotational speed because they do not rely on fan cooling but are kept low by other peripherals (for example, our lab's servers are placed in air-conditioned rooms all year round).
The second column of temp: is the temperature, unit Celsius.
The third column of perf: is the performance state, from P0 to P12,P0 represents the maximum performance, P12 represents the state minimum performance.
PWR at the bottom of column four: The power consumption, above the persistence-m: is the state of the continuous mode, although the continuous mode consumes a lot of energy, but when the new GPU application starts, it takes less time, which shows the state of off.
The Bus-id in column Five is something that involves the GPU bus, domain:bus:device.function
The DISP.A in column six is display Active, which indicates whether the GPU displays are initialized.
The memory usage below the fifth Sixth column is the memory usage.
The seventh column is the floating GPU utilization.
Above the eighth column is something about ECC.
The compute m below the eighth column is the calculation mode.
The following is an indication of the memory usage consumed by each process.

Video memory usage and GPU occupancy are two different things , the graphics card is composed of GPU and video memory, and the relationship between the memory and the GPU is a bit similar to the relationship between RAM and CPU. I ran caffe code when the memory accounted for a little, GPU accounted for a lot, brother ran TensorFlow code, memory accounted for a lot, GPU accounted for less. 4.4 How to use Keras on multiple GPU cards. (not yet resolved)

The problem has not been solved yet.
Although the data can be distributed across multiple GPUs, it seems that the song distribution is just a copy of the data, in the training process, is still a GPU in operation, the other GPU's volatile gpu-util almost 0.

More Useful Links
https://keras-cn.readthedocs.io/en/latest/for_beginners/FAQ/#gpukeras
https://www.zhihu.com/question/67239897/answer/269003621

Use the following code will be an error, temporarily do not know why:

#!/usr/bin/env python
# encoding:utf-8

"" "
@version: python3.6
@author: Xiangguo Sun
@contact: sunxiangguo@seu.edu.cn
@site: http://blog.csdn.net/github_36326955
@software: Pycharm
@file: 2clstm.py
@time: 17-7-27 5:15pm "" "

import os from
keras.utils import multi_gpu_model

Import TensorFlow as TF
import keras.backend.tensorflow_backend as KTF

#进行配置 with 60% GPU
os.environ["Cuda_ Visible_devices "]=" "
config = tf". Configproto ()
config.gpu_options.per_process_gpu_memory_fraction = 0.6
session = TF. Session (Config=config)

# Set Session
Ktf.set_session (session) ...


Model=multi_gpu_model (model,gpus=2)
#compile the Model
model.compile (loss= ' categorical_crossentropy ', Optimizer= ' Adam ', metrics=[' accuracy '])

...

The error is: cannot merge devices with incompatible IDs

4.5 How Ubuntu kills processes

Ps-ef 

kill-9 PID