Voluminous a big article, there is no, these days have been tossing this thing, really no way, do not want to use Linux but, in order to Caffe, only so, install these things, encounter many problems, each problem will be tortured for a long time, probably the first time this is. Think, after the application, should still encounter a lot of problems it, but no way, Tiger!! One suggestion here is that if you want to make a big data set in the future, it is best to give Linux more space, such as imagenet, which is estimated to be 500G. In addition, please read, at least one part of the hands-on operation, to avoid unnecessary work, writing ability is limited, please forgive me.
This installation guide, suitable for 0 basis, novice operation, please do not want to spit groove!
A brief introduction: Caffe, a convolutional neural network toolkit, is similar to Alex's cuda-convnet features, but each has its own characteristics. Is the use of C + + Cuda for the bottom-level editing, Python implementation, the original is not part of Ubuntu 12, but also the great God released the Windows version, but other relevant information less, not suitable for novice use, so or Ubuntu is more suitable for beginners. Relatively
This article contains 5 parts, including:
The first part of Linux installation
Part II installation and commissioning of Nvidia Drive and CUDA Toolkit
Part III installation and testing of Caffe
Part IV installation and commissioning of Python
Part V installation and commissioning of MATLAB
- The first part of Linux installation
Linux installation, if not Linux powder, just must, forced to use it for scientific research, recommended installation into a dual system, many online methods, here I do not elaborate, installation is also a fool-like, and windows, the process is similar to the language, if the difficulty is not big enough, Can be installed in e-version, or even Japanese, German ~ ~ ~, I am installed in Simplified Chinese version, I have a total of 100G space to install Ubuntu 14.04, this version is the latest version, there is a benefit is that Can directly access Windows8.1 NTFS partition, do not have to do extra work, and support Chinese, for example: $ cd/media/yourname/Partition name/folder name, of course GUI is more convenient my partition settings are as follows: Root partition: \ 50G, Swap partition: 16G, here, I set the same as my memory, is said to be less than 16G of memory, set into memory 1.5-twice times the home partition: The remaining 34G installed, restart the computer, some people will directly into the Linux, some will direct windows, Google or Baidu solution, because I also don't know how to fix this specific. My desktop, finished, but the notebook, the Windows partition is also destroyed, and finally can only reload Windows 8.1, but because the notebook does not have the NVIDIA GPU so do not want to toss. PS: Actually to now feel space may be small, think of imagenet 137G training file, think should put home set to 300-500g above, will more appropriate bar. The next time the installation, and then changed, and now temporarily do not want to move.
PS: Today or re-installed, the home partition to expand to 500G. So the proposal really wants big data experiment small partner, also early consideration.
Ps:ubuntu access to the page is always very slow, let me feel the gap between it and windows, but why still have so many people yearn for it? The following methods can solve some of the problems of access, especially foreign sites, but encountered some call the wall of the site, such as Google's font, or there is no way, still where to turn the circle. This seems to be the mechanism of the operating system, Windows browser will ignore those errors, and Ubuntu under the browser will not stop trying. Less nonsense, the solution to some of the problems:
$ sudo apt-get install DNSMASQ
$
sudo gedit/etc/dnsmasq.confFind # resolv-file= modified to: resolv-file=/etc/resolv.dnsmasq.conf
$
sudo cp/etc/resolv.conf/etc/resolv.dnsmasq.conf
$
sudo gedit/etc/resolv.conf
Delete all domain name servers, reserved: nameserver 127.0.0.1
- Part II: Installation and commissioning of Nvidia drivers and CUDA Toolkit
PS: In fact, we can refer to Nvidia's official Cuda installation manual, very similar, 32 pages, but all in English, I just refer to this document to complete the configuration and verification work later. Https://developer.nvidia.com/rdp/cuda-65-rc-toolkit-download#linux. In general, to enter your username and password, is to download the 6.5 account. First, Verify you to have a cuda-capable GPU do the following, and then verify that the hardware supports GPU CUDA, as long as the model exists in Https://developer.nvidia.com/cuda-gpus, there is no problem
$ LSPCI | Grep-i nvidia
Second, Verify you has a supported Version of Linux
$ uname-m && cat/etc/*releaseThe emphasis is on "x86_64", which is guaranteed to be x86 architecture, 64bit system
Iii. Verify The System has GCC installed
$ gcc--versionIf not, install it first, this is required to compile Cuda Toolkit, but Ubuntu 14.04 is the default
Iv. Download the NVIDIA CUDA toolkit:https://developer.nvidia.com/cuda-toolkit Verify address: https://developer.nvidia.com/rdp/ Cuda-rc-checksums
$ md5sum <filename>For example:
md5sum Cuda_6.5.11_rc_linux_64.run, the correct md5 of this file = a47b0be83dea0323fab24ca642346351 This feeling is very important, I first installed the time MD5 did not pass, forced installation, the result is a problem, after re-downloaded and then installed again
Five, Handle Conflicting installation methods According to the official website, before the installation of the version will have a suspicion of conflict so, the toolkit and drievers before installation have to uninstall, shielding, and so on
Six, graphical Interface shutdown exit GUI, that is, X-win interface, the operation method is: simultaneously press: CTRL+ALT+F1 (F2-F6), switch to tty1-6 command line mode. To turn off desktop services:
$ sudo stop lightdm
Seven, Interaction with Nouveaunouveau is an open source graphics driver, Ubuntu 14.04 is installed by default, but it will affect the installation of Nvidia driver, so only ask him to go back to his hometown, sorry!
1. Add Nouveau to the blacklist to prevent it from starting
$ cd/etc/modprobe.d
$ sudo vi nvidia-graphics-drivers.conf
Write: Blacklist nouveau
Save and exit: wq!
Check: $ cat nvidia-graphics-drivers.conf
2. For:/etc/default/grub, add to end.
$ sudo vi /etc/default/grub
End write: Rdblacklist=nouveau nouveau.modeset=0
Save and exit: wq!
Check: $ cat/etc/default/grub
3. The operation provided by the official website:
$ sudo mv/boot/initramfs-$ (uname-r). img/boot/initramfs-$ (uname-r)-nouveau.imgThen regenerate the initrd file
$ sudo dracut/boot/initramfs-$ (uname-r). IMG $ (UNAME-R)
$ sudo update-initramfs-uThe above is the official NVIDIA command, do not know why I would suggest that Dracut is a non-existent command, perhaps the version of the problem, or the lack of any package, but it does not matter, the second command can also be done, should be the same function. (If I understand the mistake, welcome the children to teach, I will correct)
PS: In fact, this series of work, a little confused, because some commands and files do not exist. The principle of understanding, but the steps are still a little vague, but, I was done according to the above operation, the back of the problem, should deal with the past it.
Eight, installation
In view of the installation process encountered some problems and revelations, it is recommended to install the official latest version of the graphics driver, and then install CUDA, here may be cuda built-in driver is not too complete, or a little adaptability. GTX graphics drivers are as follows (Tesla version of the driver, please go to Nvidia's official website to download):
：
http://www.geforce.cn/drivers
Switch to the directory where Cuda_6.5.11_rc_linux_64.run is located, and then execute the install command:
$ sudo sh cuda_6.5.11_rc_linux_64.run
Again, the installation must be performed before the md5sum, I first installed is executed, found not the same, and then no reason it directly installed, resulting in the installation of the Sumary display driver success, toolkit and samples failed, the second in the installation is good.
What should I do if I find MD5 detection is inconsistent? Don't tease, go to nvidia re-download on the line, the Earth people know, do not infinite circulation is good ^_^!
Here will ask you all kinds of questions, basically is accept-yes-enter-yes-enter-yes-enter, in fact, let you accept the agreement, and then install the default location confirmation and so on, recruit do not customize the installation location, the default is heaven.
After installation, you will be prompted to lose four libraries: libglu.so, libx11.so, libxmu.so, libxi.so this wood has a relationship, the next step is to solve this problem.
Ix. Extra Libraries Install some necessary library files, such as: OpenGL (e.g., Mesa), GLU, GLUT, and X11 (including Xi, XMU, and GLX).
$ sudo apt-get install Freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev LIBGL1-MESA-GLX Libglu1-mesa Lib Glu1-mesa-devThis can switch to the GUI interface in the operation, otherwise those tips information, are garbled, what can not understand, but if your system is e-text, this sentence ignored. Here, to ensure that the network does not interrupt on the line, Spit groove: Damn the Ruijie certification, this Ubuntu is out to 14.04, you 12 version of the client has not come out, harm me before the 1 full-day network, is now forced to rub net family, a little want to vomit feeling!!!
Ten, the driver is finished, you can go back to the GUI interface, you can continue to play text here ...
$ sudo start lightdm
Xi. post-installation ACTIONS
This step is to verify that the installation is correct, compile and complete the following Cuda comes with the program, it is recommended to do. Environment Setup
$ export Path=/usr/local/cuda-6.5/bin: $PATH
$ export ld_library_path=/usr/local/cuda-6.5/lib64: $LD _library_path
2. (Optional) Install writable Samples
$ cuda-install-samples-6.5.sh <dir>Install to the home, after the finished can be adjusted under the GUI, mainly in front of the requirements, there will be a sample folder Nvidia_cuda-6.5_samples in the root directory of home OK. Because of the convenience of compiling the test in all aspects behind. In fact, if the previous installation Cuda drive and toolkit all normal, this step can be omitted, should be automatically established, but check the harm.
3. Verify the Installationa. Verify the driver version, in fact, the main thing is to ensure that the driver is installed properly
$ cat/proc/driver/nvidia/version
B. Compiling the Examples
$ nvcc-vNo unexpected words should be prompted, NVCC not installed, in fact, Nvidia-cuda-toolkit compiler is not installed complete, in short, according to the prompt to continue on the good
$ sudo apt-get install Nvidia-cuda-toolkitI'm still close to the 400MB file to download, it is fully automatic, so long as the network unblocked, a cup of coffee in the hand, and then you can xxx. Sadly, I was late here, next door WiFi is broken, notebook 360 WiFi connection on a will also broken, only to collect things back to the dormitory, tomorrow to continue. Look at the table, leaning, 00:03, and the clock-out time missed again today.
The next day, installed here, you can compile, switch directories to ~/nvidia_cuda-6.5_samples, memory is not a problem, you should remember it is installed in the home folder, cross the past is good, and then execute:
$ cd/home/username/nvidia_cuda-6.5_samples
$ make
C. Running the binaries run the compiled file to see the basic information and bandwidth information for the device:
$ cd/bin/x86_64/linux/release
$./devicequery$./bandwidthtest
PS: If the test appears to say that the running version of the driver and the actual driver does not match (the original text does not remember, not written down), the reason may be because the later installed Nvidia-cuda-toolkit updated the configuration file, So the configuration of the original cuda-samples or the driver has changed, so the detection can not be compiled through. Consider the following workaround:
1. Uninstalling an existing drive
$ sudo nvidia-installer--uninstall
2. Download the appropriate version of the driver and install:
：
http://www.geforce.cn/drivers
3. Re-install Cuda Toolkit
$
sudo sh cuda_6.5.11_rc_linux_64.run
Okay, here's the end of all Nvidia CUDA installations, here's how Caffe installs
- Part III installation and testing of Caffe
for the installation of Caffe strictly comply with the official website requirements to: http://caffe.berkeleyvision.org/installation.html
First, install Blas, here you can choose (Atlas,mkl or Openblas), I use the mkl here, download and install the intel® Math Kernel Library Linux * version mkl, download link is: https://software.intel.com/ En-us/intel-education-offerings, you can download the student version, apply first, and then immediately receive an email (there is the installation serial number), open according to download on the line. After the download, to extract the files to the home folder (or directly to the tar.gz file to the home folder, in order to save space, remember to delete the compressed file after installation), or other Ext4 file system. The next step is the installation process, authorizing and then installing:
$ tar zxvf cpp_studio_xe_2013_sp1_update3.tar.gz(If you are copying the compressed files directly)
$ chmod a+x/home/username/cpp_studio_xe_2013_sp1_update3-r
$ sudo./install_gui.shPS: When installed here, I encountered some episodes, first here to remind everyone, to avoid making mistakes, these are my installation, encountered the "little Bug." 1. A very 2 problem, when the start of Linux, the hand of the system Automatic Update, may just update to a key component, authorization, always invalid, this is probably the solution is to remember the update, restart, and then shut down the machine, I did not restart the time is useless.
2. Remember to leave the unpacked Studio_xe package under the home, or simply unzip it directly in the home, note that it is saved in home/
username, Here's
usernameIs your user name. This step is mainly to let the installation program in the Linux file system, in order to modify the permissions to ensure. 3. Use chmod to authorize the folder and its sub-files, the installer is install_gui.sh, it calls the install.sh, and then calls a series of files, these files must have executable permissions, so you understand ~ Follow the above steps 4. When installing, you can install to root permissions, or sudo permissions, I am installed under root permissions to avoid trouble, then you must ensure that the root password has been set, and then let you enter the time you apply for the serial number sent to you. If not set, execute:
$ sudo passwd root
5. Everything is ok! PS2: I will not tell you, my first installation, not only installed the studio XE, also installed the MKL, also installed Openblas, in order to install Openblas also installed to Gfortran compiler, because has been compiled Caffe do not pass, in fact, this is a waste of effort, are not carefully read the official website of the instructions caused. As for the performance of this centralized library, it should be mkl>openblas>atlas, the official website of the introduction also mentioned. One thing you need to do when you're done with MKL is SET BLAS: = Open in Makefile.config, which I'll write when I install Caffe in the back.
Second, the MKL and CUDA Environment Settings folder switch to/ETC/LD.SO.CONF.D, and do the following 1. Create a new intel_mkl.conf and edit it:
$ cd/etc/ld.so.conf.d
$ sudo vi intel_mkl.conf/opt/intel/lib/intel64
/opt/intel/mkl/lib/intel64
2. Create a new cuda.conf and edit it:
$ sudo vi cuda.conf/usr/local/cuda/lib64/lib
3. Complete the link operation of Lib file, execute:
$ sudo ldconfig-v
Third, install the OpenCV1. : Https://github.com/jayrambhia/Install-OpenCV, if you feel that the difficulty is not enough, you can choose the official website of the installation package: http://opencv.org/, I am here according to the Great God compiled version of the installation.
2. Switch to the folder where you saved the file, and then install the dependencies:
sudo./dependencies.sh
3. Install OPENCV, because do not know what is the difference, so install the latest version of Opencv2_4_8 Bar, have preferences can be set according to their own requirements:
sudo./opencv2_4_8.sh
Ensure the network is unblocked, because the software needs to be networked here for a long time, please wait patiently ... , so just install other dependencies on the
1. Google Logging Library (glog),: https://code.google.com/p/google-glog/, then unzip the installation:
$ tar zxvf glog-0.3.3.tar.gz
$./Configure
$ make
$ sudo make install
2. Other dependencies to ensure success
$ sudo apt-get install Libprotobuf-dev libleveldb-dev libsnappy-dev Libopencv-dev Libboost-all-dev Libhdf5-serial-dev
If an error occurs during installation, E:sub-process/usr/bin/dpkg returned an error code (1), possibly because sudo apt-get install appears unexpectedly, do not worry, you can try this workaround:
$ cd/var/lib/dpkg
$ sudo mv Info info.bak
$ sudo mkdir info
$ sudo apt-get--reinstall install libprotobuf-dev libleveldb-dev libsnappy-dev Libopencv-dev Libboost-all-dev Libhdf5-serial-dev
V. Install Caffe and Test 1. Switch to the Caffe download folder, and then do the following:
$ cp Makefile.config.example Makefile.configIt is important to modify the newly generated makefile.config file and modify "BLAS: = Mkl".
$ make all
$ make Test
$ make Runtest
Error Fixed:
1. If you are prompted: Make:protoc: Command not found, it is because PROTOC is not installed, install a bit better.
$ sudo apt-get install Protobuf-c-compiler Protobuf-compiler
2. Hint "SRC/CAFFE/UTIL/MATH_FUNCTIONS.CU": error:calling a host function ("Std::signbit") from a Globalfunction ("Caffe::sgnbit_kernel") is not allowed "
Workaround:
Modify./INCLUDE/CAFFE/UTIL/MATH_FUNCTIONS.HPP 224 Lines
Delete (note): Using Std::signbit;
modified: Define_caffe_cpu_unary_func (sgnbit, y[i] = Signbit (X[i]));
for: Define_caffe_cpu_unary_func (sgnbit, y[i] = Std::signbit (X[i]));
This method thanks Netizen: Back hot dj$998.
get author, great God
yangqing Jia
reply, the solution is as above, there is no two.
Vi. testing with the Mnist data set
CAFFE By default will be installed in $caffe_root, is to extract to that directory, for example: $ home/username/caffe-master, so the following work, the default has been switched to the working directory.
The following work is primarily to test whether the Caffe is working properly and does not perform a detailed evaluation. For specific settings, please refer to the official website: http://caffe.berkeleyvision.org/gathered/examples/mnist.html1. Data preprocessing can be downloaded with a good data set, can also be re-downloaded, I fast speed, here is lazy to download directly, the specific operation is as follows:
$ cd Data/mnist
$
sudo sh ./get_mnist.sh
2. Rebuilding the LDB file is to process the data set identified by the binary data set for Caffe, and all subsequent data, including the Jpe file, will be processed into this format
$ cd Examples/mnist
$
sudo sh ./create_mnist.shGenerate mnist-train-leveldb/and mnist-test-leveldb/folders, which contain data sets in LDB format
3. Training Mnist
$ cd Examples/mnist
$
sudo sh ./train_lenet.sh
At this point, the Caffe installation of all the steps to the end, the following is a simple set of data comparison, experiment from the mnist data set, mainly to examine the performance of the CPU and GPU under different systems. Can see the obvious difference, although the Mnist dataset is very simple, believe that the complex data set, the difference will be greater, UBUNTU+GPU is the only choice. Test platform: I7-4770K/16G/GTX 770/cuda 6.5windows8.1 on cpu:620swindows8.1 in Gpu:190subuntu 14.04 on cpu:270s Ubuntu 14.04 on GPU : 160s
- Part IV installation and commissioning of Python
1. Install some of the dependencies that Caffe must have:
$ sudo apt-get install python-numpy python-scipy python-matplotlib python-sklearn python-skimage python-h5py Python-protobuf python-leveldb python-networkx python-nose python-pandas python-gflags Cython Ipython
2. Configure the path, edit Makefile.config
Python_include: =/usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
python_lib: =/usr/local/lib
Include_dirs: = $ (python_include)/usr/local/include
Library_dirs: = $ (python_lib)/usr/local/lib/usr/lib
3. It is important to use the IDE environment and support the Ipython output in order to ensure that the programs in the Caffe notebook are supported.
- Part V installation and commissioning of MATLAB
1. Download
As the software for commercial software, please look for yourself, install learning, and make sure not to use for commercial purposes, download 24 hours to delete ...
2. Pre-Prepare Select Mathworks.Matlab.R2014a.Unix.iso-right-use disk image mount to open "
Go to the mounted virtual disc and copy all files to the Home/matlab folder
(PS: My principle is can GUI on GUI, like CMD can be referenced to execute)Copy Crack/install.jar to home/matlab/java/jar/and overwrite source files
3. Licensing the installation folder
$ chmod a+x matlab-r
4. Installation
$ sudo./instal
LOption: Do not use the Internet installation serial number: 12345-67890-12345-67890 default path:/usr/local/matlab/r2014a Activation file: license_405329_r2014a.lic copy Libmwservices.so to/usr/local/matlab/r2014a/bin/glnxa64 installation is complete, the program default boot path:
$ sudo matlab/r2014a/bin/matlab
5. Create a shortcut1. Software Center search matlab
2. Select the installation directory:/usr/local/matlab/r2014a
6. Configure Caffe
Modified files: Makefile.config
Matlab_dir: =/usr/local/matlab/r2014a
7. Compiling the caffe file used by matlab
$ make Matcaffe
Caffe Installation 2
