Depth complex network deep Complex Networks

Turn from: https://www.jiqizhixin.com/articles/7b1646c4-f9ae-4d5f-aa38-a6e5b42ec475 (please contact me if you have copyright issues)

Currently, most of the mathematics in the deep learning model is real value, recently, the University of Montreal, the Canadian National Academy of Sciences-ENERGY/materials/Communications Research Center (INRS-EMT), Microsoft Maluuba, Element AI, a number of researchers (including Cifar Senior Fellow Yoshua Bengio) published a paper on ArXiv on NIPS 2017 (held in Long Beach, USA this December), proposing a key component that can be used in complex numerical depth neural networks, which has also open source relevant research code on GITHUB 。 The heart of the machine is a summary introduction to this paper.

• Paper Address: https://arxiv.org/abs/1705.09792

• Code Address: Https://github.com/ChihebTrabelsi/deep_complex_networks

Thesis: Depth complex network (deep Complex Networks)

At present, most of the building blocks, technologies and architectures of deep learning are based on real numerical computation and characterization. However, recent theoretical analyses of recurrent neural networks and other older foundations have shown that complex numbers can have richer characterization capabilities, and can also promote robust memory retrieval mechanisms for noise. Although they have compelling properties and potential in bringing new neural architectures, the deep neural networks of complex values have been marginalized due to the lack of building blocks needed to design the model. In this study, we provide key basic components that can be used in complex numerical depth neural networks and apply them to convolutional feedforward networks. More precisely, we rely on the complex convolution, proposed the complex numerical depth neural network, the plural batch normalization, the complex weight initialization strategy, and we also in the end-to-end training scheme to experiment with them. We show that such a complex value model can achieve comparable or better performance to its corresponding real numerical model. We tested the depth complex model on some computer vision tasks and music transcription tasks using musicnet datasets to achieve the best performance at the moment.

1 Introduction

The contribution of this paper is as follows:

1. The complex batch normalization (complex batch normalization) is formalized, see section 3.4;

2. Complex weight initialization, see section 3.5;

3. The best results are achieved in the multi-instrument music transcription data Set (MusicNet), as described in section 4.2.

3 Complex building Blocks

In this section, we give the core of our research results, and set up a mathematical framework for the Deep Neural Network building module that realizes the plural value.

Figure 1: Complex convolution and residual network implementation details

3.1 Characterization of complex numbers

3.2 Plural convolution

3.3 depth and width of deep complex networks

3.4 Complex Batch Normalization

3.5 Complex weight Initialization

3.6 Complex convolution residuals network

4 Experimental results

In this section, we give the experimental results of our model in image and music classification tasks. First, we give our model architecture, and then give its results on three standard image classification benchmarks (CIFAR-10, CIFAR-100, and Svhn), as well as automatic music transcription results on the musicnet benchmark.

4.1 Image recognition

Table 1: Model schemas. S1, S2, and S3 Filters respectively refer to the number of convolution filters used at each layer of stage 1, 2, and 3. (S) Represents a small network, (L) represents a large network.

Table 2: Classification errors on CIFAR-10, CIFAR-100, and Svhn. Note that He et al. [2016] uses a 110-layer model

Figure 3: (a) the Stage 1 feature graph as the real and imaginary pairs of each input, and (b) as a feature map of amplitude and phase

Figure 4: Stage 2 and 3 feature plots as real and imaginary pairs for each input

4.2 Automatic music transcription on musicnet datasets

Table 3:musicnet Experiment. FS represents the sample rate. The Params is the total number of parameters. We give an average accuracy (AP) indicator, which refers to the area under the precision recall curve (Precision-recall curve).

Figure 5: Precision Recall curve

by me:

3.1 Representation of Complex Numbers

N feature maps such that n was divisible by 2;

Allocate The first N/2 feature maps to represent the real components and the remaining N/2 to represent the imaginary ones .

Four dimensional weight tensor W, links Nin input feature maps to Nout output feature maps and whose kernel size is M XM.

Have a weight tensor of size (NOUTXNINXMXM)/2 complex weights

3.2 Complex Convolution

Complex filter Matrix W = A + IB;

Complex vector h = x + iy

W? h = (A? x?) B? Y) + I (B? x + A y).

Use the matrix notation to represent it:

3.3 Depth and Width in deep Complex Networks

For a given layer, the number of parameters for each of the real and imaginary weights would is equal to N/2XN/2 which m EANs N2/2 when we sum both.

For a real-valued layer, it is N2.

Assuming there is a total of L-layers, the parameters of the real value are 2L times the complex value (√2).

3.4 Complex Batch Normalization

Not to be continued ...

Depth complex network deep Complex Networks