Caffe supports multi-GPU distributed computing

Caffe allows parallel computing between multiple GPU, and multi-GPU mode is "not sharing data, but sharing network". When the number of GPU on the target machine is greater than 1 o'clock, Caffe will allow multiple solver to exist and be applied to different GPU.

Vector<int> GPUs;
Get_gpus (&gpus);
if (gpus.size () > 0)
{
    caffe::setdevice (gpus[0]);
    Caffe::set_mode (CAFFE::GPU);
    Enable multiple Solver
    Caffe::set_solver_count (Gpus.size ());
}

The first solver will become Root_solver_, and other solver will become shared_solver. Root_solver_ has a large part of the franchise, specifically the following points:

1 Log information: Avoid producing multiple duplicate information, so only allow Root_solver_ output log information;

2. Test: Only Root_solver_ can be tested, should be to reduce redundancy calculation;

3. Statistical results: Only ROOT_SOLVER_ can output statistical results, the reason is the 1th.

The place to use Solver_count directly is the DataReader object in the DataLayer layer. Each datalayer has a datareader,datareader working as an asynchronous thread, and the program allows multiple datalayer to be executed at the same time, but not multiple convlayer.

PS: This article mainly refers to Physcal "from the beginning of the Zero cottage Caffe three: Global thread Manager" article.