Install Mxnet package for mnist handwritten digit recognition

I want to write a series of deep learning simple practical tutorials, using mxnet to do the implementation of the platform of the example code to explain the deep learning commonly used in some technical direction and practical examples. The main content of this series is inclined to explain practical examples, from the sample and code to learn to solve practical problems. I will default the reader has the basic knowledge of neural network and deep learning, the reader will not see the large section deduction and theoretical elaboration. Basic theory knowledge is very important, if the reader is interested in theoretical knowledge, you can see the existing deep learning tutorials to supplement and consolidate the theoretical basis, here http://deeplearning.net/reading-list/tutorials/some good theoretical tutorials, The relevant theoretical knowledge does not dwell on this. MXnet: Lightweight distributed, portable, deep learning computing platform

Mxnet is an open-source deep learning computing platform implemented by a group of smart, hardworking young computer scientists, an important part of the DMLC Distributed machine Learning Common Toolkit http://dmlc.ml/(if you know xgboosthttps://github.com/ Dmlc/xgboost the implementation of this parallel GBT should not be unfamiliar to DMLC). The advantage of mxnet is that it is lightweight, highly portable, easily distributed and parallel, and uses video memory efficiently, and is more flexible to run on mobile devices. Its code and use of the method is also concise and clear, suitable for learning combat. So interesting deep learning tool platform, everyone go to dot this github connection to it add a star plus a fork, Portal: Https://github.com/dmlc/mxnet installation Mxnet

Mxnet supports Linux,windows and Mac platforms. The main platform used in this article is Ubuntu 14.04 LTS. Note that these series of tutorials use the Cuda platform for GPU operations, and Cuda does not support the latest Ubuntu 15.10 version of the environment and compilers (mainly GCC 5.2 compilers) While writing this article, so it is strongly recommended to stick to 14.04 The LTS version is either up to version 15.04.

The installation environment can be an entity machine with an NVIDIA graphics card or a cloud server with a GPU. If you select physical machines, do not install through virtual machines, such as running virtual Linux under native Windows, because most virtual machine software does not support direct calling of native graphics cards. If you choose a cloud server, be sure to choose GPU instance such as AWS g2.2xlarge or g2.8xlarge , or terminal.com GPU instance. Note: It is known that the terminal.com runtime can change the type of the virtual machine, but the pure CPU virtual machine does not seamlessly switch to the GPU at runtime, it is recommended to choose GPU instance from the beginning.

The following installation steps refer to the official documentation: Http://mxnt.ml/en/latest/build.html#building-on-linux, this article is based on Cuda installation and actual operation slightly modified. Base-dependent Installation

Another advantage of mxnet is that it requires only a few third-party packages, which basically require only GCC compilers, Blas, and optional installation OpenCV. If you don't have Git installed, you can install it here.

sudo apt-get updatesudo apt-get install -y build-essential git libblas-dev libopencv-dev

Download mxnet

git clone --recursive https://github.com/dmlc/mxnet

Note here that you must not forget --recursive the parameters, because Mxnet relies on the DMLC Generic toolkit http://dmlc.ml/, --recursive parameters can be automatically loaded mshadow and other dependencies. Don't worry about compiling here for the time being, we have to pack cuda. Cuda Installation

The Cuda installation method mentioned here also applies to deep learning packages other than mxnet. We download the Cuda Driver and toolkit via the official NVIDIA link, please go to https://developer.nvidia.com/cuda-downloads to select the appropriate installation method. Domestic readers recommend the network installation method deb(network) , so that Ubuntu will choose the nearest domestic source installation, the speed may be relatively fast.

If you use Ubuntu 14.04, do not go to the official website, directly run the following commands can also be called official website download (installation package larger need to wait patiently):

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1404/x86_64/cuda-repo-ubuntu1404_7.5-18_amd64.debsudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.debsudo apt-get updatesudo apt-get install cuda

If everything is installed successfully, nvidia-smi you can use the command to view your graphics card usage, the General idle video card status is like this:

The graphics card model depends on the individual economic ability, but the mxnet memory utilization is high, generally a 4G graphics card is enough to deal with most of the tool pack for a lot of memory problems.

Optional installation : Mxnet is also supported cuDNN , it is Nvidia launched the Deep Learning Accelerator toolkit, can efficiently achieve some of the convolution and other deep learning common operations, in memory usage and calculation speed can be improved. You can go here HTTPS://DEVELOPER.NVIDIA.COM/CUDNN apply for developer project, if approved by can download install CUDNN Toolkit, please refer to Nvidia official tutorial for details. Compiling a GPU-enabled mxnet

Mxnet needs to open a compile and link option to support Cuda. Locate the subdirectory in the directory that you obtained in the previous step git clone mxnet/ , copy the directory to the mxnet/make/ config.mk mxnet/ directory, open it with a text editor, and locate and modify the following three lines:

USE_CUDA = 1USE_CUDA_PATH = /usr/local/cuda

The second line is the Cuda installation directory. If you choose the default installation method, it will be in the /usr/local/cuda /usr/local/cuda-7.5 original installation directory or similar, if it is a custom directory installation, please modify this article yourself.

Additional modifications are required if the user chooses to install other Blas implementations such as Atlas or Openblas. If Ubuntu is atlas implemented ( sudo apt-get install libatlas-base-dev or sudo apt-get install libopenblas-dev ), it needs to be modified to:

USE_BLAS = atlas 或者 openblas

After the modification, mxnet/ compile in the directory (which -j4 is an optional parameter that is compiled with 4 threads):

make -j4

Note: If there are no Cuda-supported graphics cards (such as Intel's Iris graphics card or AMD's R-series graphics card) or no graphics card, installing and compiling GPU versions of Mxnet errors. The workaround is to USE_CUDA = 1 change back and USE_CUDA = 0 ensure that USE_OPENMP = 1 mxnet automatically compiles the CPU version and uses OpenMP for multi-core CPU calculations. Depending on the problem, the GPU version typically has about 20-30 times the speedup compared to the CPU version. Installing Python support

Mxnet supports Python calls. In simple terms, install it:

cd python; python setup.py install

It is recommended to use Python version 2.7, which requires pre-installation setuptools and numpy ( sudo apt-get install python-numpy ). If your system has some difficulty installing numpy, consider installing a python distribution such as Anaconda or Miniconda:

wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.shbash Miniconda-latest-Linux-x86_64.sh（确认回答若干安装问题后）conda install numpy

Run mnist handwritten digit recognition

November 19, 2015 update : The sample here is based on the old version mxnet/example of the directory structure, the new version of the Mnist code mxnet/example/image-classification/ below, you can turn on the --gpu (gpu_id) GPU computing options, please update yourself and see the new instructions: https:// Github.com/dmlc/mxnet/tree/master/example/image-classification.

When mxnet everything is installed, you can try the simplest example, mnist handwritten digit recognition. The Mnist dataset contains a training dataset of 60,000 handwritten digits and 10,000 test datasets, each of which is a grayscale image of 28x28. mxnet/example/mnistyou can find a sample of Mxnet's own mnist, which we can run first:

cd mxnet/example/mnistpython mlp.py

mlp.pyAutomatically downloads the mnist data set and waits patiently for the first time it runs.

Note : mlp.py using the CPU by default, the training process can run but is slow. We have already installed the GPU and just need to modify one line of code to FeedForward change the CPU portion of the call to GPU so that the mxnet can run on the GPU:

model = mx.model.FeedForward(        ctx = mx.cpu(), symbol = mlp, num_epoch = 20,        learning_rate = 0.1, momentum = 0.9, wd = 0.00001)   

Become:

model = mx.model.FeedForward(        ctx = mx.gpu(), symbol = mlp, num_epoch = 20,        learning_rate = 0.1, momentum = 0.9, wd = 0.00001)   

Is it much quicker to run again? The advantage of mxnet is the simplicity of the interface. At run time, the nvidia-smi command to view the video card status is almost like this:

You can see that the python process is using the GPU, because this is a relatively small problem while mxnet memory optimization, GPU utilization between 30% to 40%, video memory occupies 67MB. Problems that may occur

You may encounter this problem when you run the GPU example:

This is because the Cuda dynamic link library is not added to the path, and the workaround is to ./bashrc include:

export LD_LIBRARY_PATH=/usr/local/cuda-7.5/targets/x86_64-linux/lib/:$LD_LIBRARY_PATH

Or when compiling the mxnet, in config.mk the

ADD_LDFLAGS = -I/usr/local/cuda-7.5/targets/x86_64-linux/lib/ADD_CFLAGS =-I/usr/local/cuda-7.5/targets/x86_64-linux/lib/

Mnist Code Simple explanation: Design one of the simplest multilayer neural networks

mlp.pyThe implementation is a multilayer Perceptron network (Multilayer Perceptron (MLP)) or a multilayer neural network. In Mxnet, implementing a MLP first requires defining the structure of the MLP, such as the one in the code for a three-layer network:

data = mx.symbol.Variable(‘data‘)fc1 = mx.symbol.FullyConnected(data = data, name=‘fc1‘, num_hidden=128)act1 = mx.symbol.Activation(data = fc1, name=‘relu1‘, act_type="relu")fc2 = mx.symbol.FullyConnected(data = act1, name = ‘fc2‘, num_hidden = 64)act2 = mx.symbol.Activation(data = fc2, name=‘relu2‘, act_type="relu")fc3 = mx.symbol.FullyConnected(data = act2, name=‘fc3‘, num_hidden=10)mlp = mx.symbol.Softmax(data = fc3, name = ‘mlp‘)

Simply explain these lines of code: Mnist datasets Each set of data is a grayscale image of 28x28, almost like:

Each set of data can be represented as a one-dimensional array of length 28x28=784, and each element of the array is a grayscale value for that pixel. Each layer of MLP needs to define the style of this layer of nodes, such as fc1 accepting the first layer of input, which is defined as an all-link layer mx.symbol.FullyConnected , data which contains 128 nodes () by accepting input num_hidden . Each layer also needs to define activation functions Activation , such as the activation function between the first and second layers relu (representing rectified linear unit or called Rectifier ) Relu is the most common activation function in the deep neural network, Mainly because the calculation function is relatively easy and the gradient descent does not diverge, and because the mnist problem is more sparse and more suitable for relu. The main purpose of this space is to introduce a network, refer to Wikipedia for background information about Relu, and other relevant tutorials. The second layer of the network is fc2 similar to the first layer, and it accepts fc1 the data as input and outputs it to the third layer. The third layer is fc3 similar to the first two, unlike the result output layer, which produces the probability that the input image corresponds to each number in a total of 10 digits, so it's num_hidden=10 . 0-9.

After designing the network structure, mxnet need to declare the input feature format, because each picture is 28x28 size, according to the gray value of each pixel expands into a column vector is 784 dimensions, we can tell mxnet data input size is 784, mnist_iterator is a python generator provides 100 sets of data at a time to the MLP we have just designed, see the same directory data.py :

train, val = mnist_iterator(batch_size=100, input_shape = (784,))

Next let Mxnet build and run this model, it is so simple, if you will scikit-learn feel very gracious, right (remember the line that the specified GPU has just been modified to run?) ）：

model = mx.model.FeedForward(        ctx = mx.gpu(), symbol = mlp, num_epoch = 20,        learning_rate = 0.1, momentum = 0.9, wd = 0.00001)   model.fit(X=train, eval_data=val)

Here, you will basically implement a multilayer perceptron MLP, congratulations on this is the first step in mastering deep learning. Mxnet is much simpler than writing a configuration file for other tools such as Caffe. Most of the practical problems of deep learning in industry and academia revolve around designing multilayer perceptron, and there are many interesting problems in the design of structure design activation function.

Some readers will ask if MLP is not designed like mnist handwritten numeral recognition. No, this three-tier network is just one of the simplest examples of MLP, where each layer does not necessarily need this. Designing a better and more efficient multilayer neural network is no more than art. For example, in mnist the same directory lenet.py is the Yann LeCun design convolution network to achieve digital recognition, each layer of network needs to do is Convolution Activation and Pooling (if you want to know what these three specific, see his deep learning tutorial, It may also be mentioned in future articles.

As after-school homework, the reader can try to adjust mlp.py the number of different nodes and activation function to see what the number of digital recognition increase, you can also increase the num_epoch adjustment learning_rate parameters, in the forward, comment or message write your design methods and recognition accuracy (and no reward, eh). Kaggle for mnist Data set there is a teaching contest, the reader can use mxnet training a own mnist model, the results submitted to a ratio, remember that you are doing with mxnet yo, Portal: Https://www.kaggle.com/c/digit-recognizer PostScript

This article is the first in this series, I meant to write a mxnet GPU installation method, and then think of an example to explain the various models by the way as an additional depth learning introductory tutorial. Some of the following articles will pick Mxnet's own examples and introduce some common interesting deep learning models, such as RNN,LSTM, and their implementations in mxnet, such as writing an automatic lyrics machine to imitate Wang Feng's lyrics. Mxnet so interesting deep learning tool platform, everyone go to this GitHub connect to it add a star to add a fork, Portal: https://github.com/dmlc/mxnet

Install Mxnet package for mnist handwritten digit recognition