Operation was explicitly assigned to /job:ps/task:0/device:CPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]

標籤：assigned   ida   lidar   flag   原因   names   ted   exp   error:   

訓練啟動時報錯：

tensorflow.python.framework.errors_impl.InvalidArgumentError: Cannot assign a device for operation ‘save/RestoreV2_10‘: Operation was explicitly assigned to /job:ps/task:0/device:CPU:0 
     but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]. Make sure the device specification refers to a valid device.
[[Node: save/RestoreV2_10 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:ps/task:0/device:CPU:0"](save/Const, save/RestoreV2_10/tensor_names, save/RestoreV2_10/shape_and_slices)]]

原因：available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]表示tf.Session沒有串連到tf.train.Server，In particular, it seems to be a local (or "direct") session that can only access devices in the local process.

解決辦法：要解決這個問題，需要在建立session時添加server.target。例如:

# Creating a session explicitly.with tf.Session(server.target) as sess:# ...# Using a `tf.train.Supervisor` called `sv`.with sv.managed_session(server.target):# ...# Using a `tf.train.MonitoredTrainingSession`.with tf.train.MonitoredTrainingSession(server.target):# ...

 

我們的代碼

with tf.train.MonitoredTrainingSession(  # is_chief=is_chief,  checkpoint_dir=checkpoint_dir,  save_checkpoint_secs=FLAGS.save_interval_secs,  save_summaries_steps=100,  save_summaries_secs=None,  config=sess_config,  hooks=hooks) as sess:

 

MonitoredTrainingSession中沒有指定：

master=server.target

最終代碼是：

with tf.train.MonitoredTrainingSession(  master=server.target,  is_chief=is_chief,  checkpoint_dir=checkpoint_dir,  save_checkpoint_secs=FLAGS.save_interval_secs,  save_summaries_steps=100,  save_summaries_secs=None,  config=sess_config,  hooks=hooks) as sess:

　　

 

參考： https://stackoverflow.com/questions/42397370/distributed-tensorflow-save-fails-no-device

 

Operation was explicitly assigned to /job:ps/task:0/device:CPU:0 but available devices are [ /job:localhost/replica:0/task:0/cpu:0 ]